RFC: Support for external observability providers - Metrics

[roger-zhangg]

Is this related to an existing feature request or issue?

#1433
#2014

Which AWS Lambda Powertools utility does this relate to?

Metrics

Summary

Problem

Customers has been using Lambda Powertools alongside with third party observability provider for Logging, metrics and tracing. Focusing on metrics part (this RFC is a part of support for observability provider. For Logging please check out this RFC), while Powertools for AWS Lambda (Powertools) provided powerful and easy to use metrics class. However, Powertools only supports AWS CloudWatch Embedded Metric Format (EMF), which is unfriendly to third party observability provider, bringing hardship to our customers when trying to ingest this format into other observability solutions, such as DataDog.

Goal

Goal for this RFC is to enable third party observability provider like Datadog, NewRelic, Lumigo, etc. to create their own Powertools specific metric provider and offer to their own customers. e.g., DataDog metric format. And if customer eventually want to add additional value or add features themselves. They can easily do so by overriding a provider too.

Use case

Typical use case of this is utility would be for customers who use Lambda Powertools to collect metrics and want to have them ingested into a third party observability provider.

Proposal

Current metric experience

The current Powertools’ metrics utility creates custom metrics asynchronously by logging metrics to standard output following Amazon CloudWatch Embedded Metric Format (EMF), and these metrics can then be seen in the CloudWatch console. This utility can aggregate up to 100 metrics using a single CloudWatch EMF object (which is in JSON format).

from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit
from aws_lambda_powertools.utilities.typing import LambdaContext


# Current experience in using metrics should not change, i.e. these metrics would still output to CW
metrics = Metrics()

@metrics.log_metrics # ensures metrics are flushed upon request completion/failure
def lambda_handler(event: dict, context: LambdaContext):
    metrics.add_metric(name="SuccessfulBooking", unit=MetricUnit.Count, value=1)

# JSON output
#     "_aws": {
#         "Timestamp": 1656685750622,
#         "CloudWatchMetrics": [
#             {
#                 "Namespace": "ServerlessAirline",
#                 "Dimensions": [
#                     [
#                         "environment",
#                         "service"
#                     ]
#                 ],
#                 "Metrics": [
#                     {
#                         "Name": "TurbineReads",
#                         "Unit": "Count"
#                     }
#                 ]
#             }
#         ]
#     },
#     "environment": "dev",
#     "service": "booking",
#     "TurbineReads": [
#         1.0,
#         8.0
#     ]
# }

Metric provider proposal

For this new utility, we propose a new metrics class for observability providers. With an optional parameter provider to allow developers to pass in observability provider pre-defined or user custom provider. And the output will be in observability provider friendly format. The below code snippet is a rudimentary look at how this utility can be used and how it will function.

The default use case for metrics before is metrics=Metrics()
After we have this provider feature, Customers can still use original CloudWatch Metrics by metrics=Metrics() or metrics=CloudWatchEMF(). They can also use provider for third party provider by e.g.:Metrics=DataDogMetrics()

from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit
from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.metrics.provider import DatadogMetrics
from aws_lambda_powertools.metrics.provider import DatadogMetricsProvider

# Use datadog-defined metrics provider
provider = DatadogMetricsProvider()
metrics = DatadogMetrics(provider=provider)

@metrics.log_metrics
def lambda_handler(event: dict, context: LambdaContext):
    metrics.add_metric(name="SuccessfulBooking", value=1, tags={"product":"ticket","ordier":"online"})
    
# JSON output to log
# {
#     "m": "SuccessfulBooking", 
#     "v": 1,
#     "e": 1678509106, 
#     "t": "['product:ticket', 'order:online']"
# }

self-defined metrics provider usage

If the customer would like to use another observability provider, or define their own metrics functions, we will define an interface that the customer can implement and pass in to the Metrics class provider parameter

from aws_lambda_powertools.metrics.provider import MetricsBase, MetricsProviderBase


class DataDogProvider(MetricsProviderBase):
    def __init__(self, namespace):

    def add_metric(self, name: str, value: float, timestamp: Optional[int] = None, tags: Optional[List] = None):
        self.metrics.append({"m": name, "v": int(value), "e": timestamp, "t": tags})

    def serialize(self) -> Dict:
        # logic here is to add dimension and metadata to each metric's tag with "key:value" format
        extra_tags: List = []
        output_list: List = []

        for single_metric in self.metrics:
            output_list.append(
                {
                    "m": f"{self.namespace}.{single_metric['m']}",
                    "v": single_metric["v"],
                    "e": single_metric["e"],
                    "t": single_metric["t"] + extra_tags,
                }
            )
        return {"List": output_list}

    def flush(self, metrics):
        # flush

    def clear(self):


class DataDogMetrics(MetricsBase):
    """Class for datadog metrics standalone class.

    Example
    -------
    dd_provider = DataDogProvider(namespace="default")
    metrics = DataDogMetrics(provider=dd_provider)

    @metrics.log_metrics(capture_cold_start_metric: bool = True, raise_on_empty_metrics: bool = False)
    def lambda_handler(event, context)
        metrics.add_metric(name="item_sold",value=1,tags)
    """

    # `log_metrics` and `_add_cold_start_metric` are directly inherited from `MetricsBase`
    def __init__(self, provider):
        self.provider = provider
        super().__init__()

    def add_metric(self, name: str, value: float, timestamp: Optional[int] = None, tags: Optional[List] = None):
        self.provider.add_metric(name=name, value=value, timestamp=timestamp, tags=tags)

    def flush_metrics(self, raise_on_empty_metrics: bool = False) -> None:
        metrics = self.provider.serialize()
        if not metrics and raise_on_empty_metrics:
            warnings.warn(
                "No application metrics to publish. The cold-start metric may be published if enabled. "
                "If application metrics should never be empty, consider using 'raise_on_empty_metrics'",
                stacklevel=2,
            )
        self.provider.flush(metrics)
        self.provider.clear()
         
    
metrics = DataDogMetrics(provider=DataDogProvider())

@metrics.log_metrics
def lambda_handler(event: dict, context: LambdaContext):
    metrics.add_metric(name="SuccessfulBooking", value=1, tags={"product":"ticket","ordier":"online"})
    # call functions in provider
    metrics.provider.add_tag({"test":True})       

# JSON output to log
# {
#     "m": "SuccessfulBooking", 
#     "v": 1,
#     "e": 1678509106, 
#     "t": "['product:ticket', 'order:online']"
# }

Out of scope

Introduction of third party observability provider dependency to submit metrics to API is out of scope. The configuration included in this project should only support modifying the metrics outputted to log. But on their end, customer can still submit to API if they implement observability provider SDK in metrics' flush function

Potential challenges

How to provide more value to customer with current Metrics provider design

• Currently we just provided log_metrics decorator and _add_cold_start_metric function in the base metrics provider function.
• Other than these two are just basic framework of metrics class.
• What else should we include in the metrics class to provide more value to our customer?

Dependencies and Integrations

Dependencies

No additional dependencies required

Changes and Additions to Public Interfaces

• Modification in Metrics, MetricsManager Class
  • Add an optional provider parameter in metrics class.
  • Refactor the Metrics Class to enable use of provider. But all currently exist function will have same output.
  • Provide a AWS CloudWatch EMF Metrics provider as default metrics provider.

Performance Impact

Little or no performance impact expected

Backwards Compatibility and Upgrade Path

Full backward compatibility, old metrics codes without provider parameter will get exact same output as before.

Alternative solutions

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[heitorlessa]

Thanks a lot @roger-zhangg for taking the time in creating this! I'm so excited we're kicking the observability provider discussion.

I like the general direction but the serializer idea will likely limit us in validating custom metric format, limits, etc.

For example, here are some initial questions:

• How can a metric provider enforce their metric limits? e.g., EMF supports 100 per blob, there could be other limits in another provider
• How can a customer use custom provider metric units?
• How can a provider validate metric units to return an actionable error of valid values?
• Is there anything we should learn from OpenTelemetry Metrics API/Sink, Datadog metrics?

If we think in terms of a provider who also brings their serializer, we might be able to solve this:

Nothing set on stone, just throwing ideas atm

from numbers import Number
from aws_lambda_powertools.metrics.providers import MetricsProvider


class DatadogMetricsProvider(MetricsProvider):
    METRIC_LIMIT = 200  # super().add_metric will use that value to validate, etc.

    def serialize(metrics: MetricSummary):
        # convert metrics into custom format
        ...
    
    def validate(metrics: MetricSummary) -> bool:
        # validate metric schema
        ...

---

Ad-hoc comments

If serializer is provided with list of serializers, Metrics class will serialize metrics with every serializer and return appended result. This utility will allow our user to output several different formats of metrics at the same time.

Let's validate with customers before we implement this one out. I can see why a customer would want to have metric duplicated in N providers, however, this gets expensive really quickly making it often unfeasible. Please, do correct me if I missed or misinterpreted the intent.

Potential challenges: Not all observability provider support extract metric from log

This is the core of the RFC that needs some further thinking. A Provider mechanism might be more powerful to extend later.

In addition, we will provide a pre-defined DataDog format metric serializer.

For now, let's use as a reference to test but not commit to make Datadog provider as part of our source code. We need to circle back with Lambda and Partner teams, so to not get accidentally caught in a race to add the next Observability Partner and accidentally manage tech debt from a 3rd party. OpenTelemetry format, however, would be a good goal.

[roger-zhangg]

Thanks for the comment Heitor!

Some thoughts on questions

• How can a metric provider enforce their metric limits

  • Currently Datadog only parse 1 metrics from 1 log. Thinking it more broadly, We can have a parameter to control maximum metrics in one log.(like 1 for datadog)

• How can a customer use custom provider metric units?

  • For this if other provider doesn't have a metrics unit, it would depend on custom's preference to either merge the unit into metrics name or tags.

• How can a provider validate metric units to return an actionable error of valid values?

  • Validate would be a good idea.

• Is there anything we should learn from OpenTelemetry Metrics API/Sink, Datadog metrics

  • Previously I'm thinking about compatibility with AWS EMF, taking OpenTelemetry into our scope would changes a lot.

• Potential challenges: Not all observability provider support extract metric from log

  • For this part, There are two possible solutions
    • User can implement observability provider sdk in custom serializer to make it into a log submitter
    • User can use manual flush and add observability provider sdk there

Further thinking

Please correct me if I'm wrong. From my perspective, this custom serializer solution would give our customer more freedom on how they want to serialize/process their metrics. I don't think this would contradict with the provider solution you've proposed. They could be two individual ways to define the metrics process(also can work together?).

For example, if we have a pre-defined provider class. But our user wants some custom conversion on the metrics before submit. They could still utilize the custom serializer along with the provider.

I wonder would it be more smooth to provide the serializer option first and works on the provider part once we have more data?

[heitorlessa]

I definitely see where you're coming from. From the surface, a serializer would be easy enough to maintain and to use. However, this area is much deeper as you start digging into the ecosystem, how customers use, extend, and how non-developers cognitive load apply.

To give you an example from previous scars in Logger and Event Handler features.

At a first glance, serializer would solve the immediate need if all we cared was Datadog or NewRelic. The minute you add OpenTelemetry, or additional partners and custom solutions like ELK, we accidentally go down the road of having multiple parameters and exceptions. It gets worse when you factor the different ways one handles metric resolution, metric units, metric validation, high-cardinality, tags/dimensions, validation - it's almost impossible to predict what customers will need next, therefore adding another parameter will accidentally make other customers decision making process harder.

Overall, this increases cognitive load, and we lose the value of Powertools that customers love that kicked off this work: "As a customer, I want to use the X Observability Provider but keep Powertools straightforward UX".

Provider approach

If we shift towards a Provider approach, we can now: (1) Create a specification of what the behaviour of a provider should be, (2) Create a CloudWatch EMF Provider from the new specification, and (3) Refactor Metrics feature to primarily depend on a given Provider (e.g., provider.add_metric, provider.serialize()) while keeping Powertools features as the value add. If no provider is set, we default to our new CloudWatch EMF Provider making it seamless and requiring no code change from anyone using (or that will use) our Metrics feature 🎉.

This opens the door for external providers like Datadog, NewRelic, Lumigo, etc. to create their own Powertools specific provider and offer to their own customers. If customers eventually want to add additional value add features themselves - e.g., company policy is to add a metric for each application cost center, they can easily do so by overriding a provider too 🎉.

PS: I wrote this before our 1:1 call. Please do shout out if there are any areas you want me to clarify - THANK YOU for helping out!

[roger-zhangg]

Thanks Heitor, I've updated the RFC accordingly.

[heitorlessa]

Re-reading...

[heitorlessa]

Looks so much better, thanks a lot @roger-zhangg! Answering the questions you left out, please let me know if I missed any.

Potential challenges

Currently the Metrics class is initiated with namespace and dimension parameter. But these two parameter might be useless to other provider.

We could add a new context attribute and a inject_context method in the base provider, which we would call at initialization (late-binding). This means we can propagate information to the provider, typed even, they could do as they please, and could even ask us to expose more information later that won't impact any existing provider out there.

Let the provider and the customer own what goes into the args/kwargs, not us (dependency inversion)

from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit
from aws_lambda_powertools.utilities.typing import LambdaContext
from datadog_lambda.metrics import DatadogLPMetricsProvider

# Use datadog-defined metrics provider
datadog = DatadogLPMetricsProvider(flush=True, flush_limit=100)
metrics = Metrics(provider=datadog)

@metrics.log_metrics
def lambda_handler(event: dict, context: LambdaContext):
    metrics.add_metric(name="SuccessfulBooking", unit=MetricUnit.Count, value=1)

Metrics init

class Metrics(...):
    def __init__(self, ...):
        self.namespace = resolve_namespace(...)
        self.provider = provider

        self.provider.inject_context(context={"namespace": self.namespace, ...})

Datadog provider

class DatadogLPMetricsProvider(MetricsProvider):
    def serialize(metrics: MetricsSummary):
        app_name = self.app_name or self.context.get("namespace")
        ...

Currently the Metrics class has add_namespace and add_dimension function that might be useless to other provider

It's within the provider responsibility to repurpose data - e.g., add_dimension could simply call add_tag in a Datadog provider for example.

If they don't repurpose it, we will send that information for them to be handled during serialize call either way :-)

How to define boundary between LP metrics and provider

What process should be done in metrics class (common metrics process, but what is common?)

That's data that I'd expect from your research so we could discuss it. For example, besides Metrics, how do other providers handle Dimensions (datapoint ownership) and High-cardinality (metadata)?

Answering this question is key to derive a proposal for a base provider, and figure out if we need to rethink minimum responsibilities beyond serialize.

Take for instance areas that aren't common: metric tag in Datadog. In CloudWatch, the closest is dimensions. Both of them are key-pair values tag_name:tag_value, both seem to accept only string values, but their serialized output is different -- here, we continue with add_dimension responsibility, Datadog provider could optionally add a DIMENSION_LIMIT class variable to be automatically validated at add_dimension call, and only care about overriding serialize to change the dimension format into a tag format. They could go the extra mile and create a metric schema, then override validate to raise an exception if there's a minimum/maximum length for tag_name or tag_value for example.

what should be implemented in provider class?

serialize. Why? Because here's how I'd break it down:

• Add metrics. Typically unnecessary. We already handle this nicely and they could even add a METRIC_LIMIT class variable that we would honour the upper limit.
  • When to override? When they have additional logic that must happen for every added metric.
• Flush. Typically unnecessary. We already flush to standard output which is a common source for existing integrations already - for example, all AWS Observability Partners can extract metrics async from CloudWatch Logs.
  • When to override? When they must communicate with a local daemon running as a Lambda Extension. Rare exceptions would be a provider wanting to push directly to an endpoint but that increases customers cost and failure modes.
• Validate. It'd be a good practice but if they don't have a schema then this becomes unnecessary boilerplate. From our side, it'd be just a callable that would return True if not implemented.
  • When to override? When they have a schema they want to validate metric values, units, etc. Alternatively, they could override add_metric, add_dimension, etc and validate early.