Feature request: support high resolution metrics

[dreamorosi]

Use case

Amazon CloudWatch announced support for high resolution metric extraction from structured logs (EMF). Customers can now provide an optional StorageResolution parameter within the EMF specification with a value of 1 or 60 (default) to indicate the desired resolution (in seconds) of the metric.

We should consider adding support for this new optional parameter to Metrics.

The EMF log should have this format:

{
  "_aws": {
    "CloudWatchMetrics": [
      {
        "Metrics": [
          {
            "Name": "Time",
            "Unit": "Milliseconds",
            "StorageResolution": 60 // <- new key
          }
        ],
        ...
      }
    ]
  },
  "Time": 1
}

As part of this issue we should also update the API docs, documentation, and unit/integration tests.

Solution/User Experience

Proposed DX:

import { Metrics, MetricUnits, MetricResolutions } from '@aws-lambda-powertools/metrics';

const metrics = new Metrics();

export const handler = async (_event: any, _context: any): Promise<void> => {
    // Publish a metric with standard resolution i.e. StorageResolution = 60
    metrics.addMetric('successfulBooking', MetricUnits.Count, 1, MetricResolutions.Standard);
    // Publish a metric with high resolution i.e. StorageResolution = 1
    metrics.addMetric('failedBooking', MetricUnits.Count, 1, MetricResolutions.High);

    // The last parameter (storage resolution) is optional
    metrics.addMetric('successfulUpgrade', MetricUnits.Count, 1);
};

To support the proposal, we should add a new MetricResolutions enum that looks similar to this:

enum MetricResolutions {
  Standard = 60,
  High = 1,
}

supported by types like:

type MetricResolutionStandard = MetricResolutions.Standard;
type MetricResolutionHigh = MetricResolutions.High;

type MetricResolution = MetricResolutionStandard | MetricResolutionHigh;

export {
  MetricResolution,
  MetricResolutions,
};

as well as add a new optional parameter to the Metrics.addMetric() method, which would change its signature to:

public addMetric(name: string, unit: MetricUnit, value: number, storageResolution?: MetricResolution): void

Note
The EMF specification states that if a value is not provided, then a default value of 60 is assumed (aka standard resolution). For this reason, if the storageResolution parameter is not specified we won't add the resolution.

The change will likely also require modifying the logic of the Metrics.storeMetric private method to allow the new optional field.

Alternative solutions

No response

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Python, Java

[niko-achilles]

@dreamorosi i can contribute to this new feature .

[dreamorosi]

Thank you Niko, looking forward to collaborate on this one.

Before starting with the implementation it'd be interesting to give some thought on whether storageResolution is the best name for the new parameter. In my proposal I used that name because of how it's called in the EMF specification, however thinking forward to when we might support other providers we should choose a name that fits with most. Can we research if this type of parameter exists in other providers, and if so have a discussion on how to call it? For now I'd start with the main observability providers like Datadog, Prometheus, New Relic, etc. (suggestion from @heitorlessa)

[niko-achilles]

@dreamorosi and @heitorlessa given a DX proposal is expressed in the use case description ,
and taking under consideration as mentioned in the comment about supporting providers ,
a research about the parameter name is an efficient cause for word choice.

I have some questions for the research.

StorageResolution is a property name of the EMF Data Model, noun phrase, group of nouns Storage and Resolution. Refers to Decision/Action. The name of that property reflects that CloudWatch stores the metric with... resolution .

• Storage: the act of putting things in a special place for use ... , see dictionary definition

• Resolution: the act or process of resolving, see dictionary definition

The research is about the parameter name: storageResolution in the function signature

public addMetric(name: string, unit: MetricUnit, value: number, storageResolution?: MetricResolution): void

The research should answer the following questions in steps:

Is the parameter name storageResolution based on meaning ?
Are other providers chosen a word based on meaning ? is the meaning the same as cloudwatch concept ?
Do observability specifications exist in this field of Metrics ? What word have the authors of the specification chosen ? Is the word based on meaning ?
Is the parameter name storageResolution descriptive enough to use with its type to determine the meaning in usage scenarios ?
Does the type MetricResolution provide assistance to a developer as guide for allowed values ?
Is the plural phase MetricResolutions of the enum descriptive to choose the right value or decide to use the default value ?

[dreamorosi]

Hey Niko, thanks for breaking down the topic.

I would say these two:

Are other providers chosen a word based on meaning ? is the meaning the same as cloudwatch concept ?
Do observability specifications exist in this field of Metrics ? What word have the authors of the specification chosen ? Is the word based on meaning ?

As you correctly point out, the current storage resolution comes from the EMF specification, which we support today, so we might still need/want to stick to it.

I think at this stage we just wanted to investigate if there's any similar parameter that covers a similar concept in other providers, but just for reference, Powertools for Python chose to go with the EMF parameter name and already made a release earlier today, so maybe we can do the same?

[niko-achilles]

I think at this stage we just wanted to investigate if there's any similar parameter that covers a similar concept in other providers, but just for reference, Powertools for Python chose to go with the EMF parameter name and already made a release earlier today, so maybe we can do the same?

Andrea, yes of course . because storageResolution is based on meaning, and also pascalCase which is a Typescript style

• meaning: CloudWatch stores the metric with... resolution

Should i continue with these questions ?

• Is the parameter name storageResolution descriptive enough to use with its type to determine the meaning in usage scenarios ?
• Does the type MetricResolution provide assistance to a developer as guide for allowed values ?
• Is the plural phase MetricResolutions of the enum descriptive to choose the right value or decide to use the default value ?

[heitorlessa]

Please allow me to take time to respond this next week, as soon as I have a break.

@leandrodamascena had a quick research on similar systems (e.g., Prometheus), and "resolution" parameter name was the closest "neutral" to this.

When we get to scope providers, we will have more thorough discussions and Tenets, as to not fall into the trap of optimising for the lowest common denominators and accidentally back ourselves into a corner we can't innovate/disagree.

Have an excellent and joyful weekend everyone!

[niko-achilles]

Please allow me to take time to respond this next week, as soon as I have a break.

Hi Heitor, i am commenting because i invested time early in the morning, the main trigger is curiosity . This additional comment may bring things together for further discussion with Andrea and Leandro.

@leandrodamascena had a quick research on similar systems (e.g., Prometheus), and "resolution" parameter name was the closest "neutral" to this.

Is the research based on meaning ? Because the motivation, usage scenario and end cause of resolution in Prometheus is a different one to my understanding in comparison to an act with meaning of storage and resolution .
Reference Link: https://prometheus.io/blog/2019/01/28/subquery-support/

Can we look at the current research of Leandro, for learning ?
Furthermore i liked this article about Relationship between Flush Intervals in StatsD and the Time Resolution of Metrics in the Database, link: https://hostedmetrics.com/articles/introduction-to-statsd/

and also looked how aws-embedded-metrics-node uses the Storage Resolution concept: https://github.com/awslabs/aws-embedded-metrics-node/blob/master/src/logger/MetricsLogger.ts#L134

When we get to scope providers, we will have more thorough discussions and Tenets, as to not fall into the trap of optimising for the lowest common denominators and accidentally back ourselves into a corner we can't innovate/disagree.

Do you mean that the approach of deriving meaning and usage scenario is the balanced efficient method ?

Additional curiosity of me by looking at the code of Metrics in powertools for typescript:
Why is the method named for publishing named publishStoredMetrics ?

In lambda given entry-point, handler and event driven, the usage scenario is publishing metrics at end of handler invocation, automatically or manually.
Depends on the mechanics used by a dev. like decorator or middleware or manual .
An interval concept has no meaning.
Then what was the rationale behind naming the method publishStoreMetrics ? Why not publishMetrics ?

https://github.com/awslabs/aws-lambda-powertools-typescript/blob/db215dd80c9a18a612c32c9fa5e373af1c110476/packages/metrics/src/Metrics.ts#L308