RFC: Support for external observability providers - Logging

[seshubaws]

Is this related to an existing feature request or issue?

#1433

Which AWS Lambda Powertools utility does this relate to?

Logger

Summary

This RFC is the first of three that will detail how the Logger, Metrics, and Tracer utilities can be extended to support a bring-your-own-observability-provider format for customers who want to use platforms other than AWS CloudWatch.

This RFC will be specifically for the Logger. Currently, in the Logger for Python (and Typescript), customers can bring their own Logging Formatter and have the option to provide a configuration option to define the output format when they're setting up their Logger in order to allow for better integration with observability solutions other than CloudWatch.

Since this is already quite powerful, we are seeking to improve the existing mechanism by offering out of the box support for Datadog and a few other providers (TBD) for the Logger Formatter.

Metrics RFC: #2015
Tracer RFC: #2030

Use case

The primary use case for this utility would be for developers who want a simplified way to integrate the Logger Formatter with the most popular third party observability providers (ie. DataDog).

Proposal

Current Logger Formatter experience

Below is a simple snippet of how the Logger Formatter currently works:

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging.formatter import LambdaPowertoolsFormatter

class CustomFormatter(LambdaPowertoolsFormatter):
    def serialize(self, log: dict) -> str:
        """Serialize final structured log dict to JSON str"""
        log["event"] = log.pop("message")  # rename message key to event
        return self.json_serializer(log)  # use configured json serializer
        logger = Logger(service="payment", logger_formatter=CustomFormatter())logger.info("hello")

logger = Logger(service="payment", logger_formatter=CustomFormatter())
logger.info("hello")

Output

{
    "level": "INFO",
    "location": "hello_name:8",
    "event": "Request from Alice received",
    "timestamp": "2023-03-21 13:25:36,129+0100",
    "service": "service_undefined"
}

Logger Formatter proposal

For this new utility, we propose a new parameter to the existing logger that developers can use to specify which observability provider they would like to receive their logs. The below code snippet is a rudimentary look at how this utility can be used and how it will function. Out of the box, we will support DataDog and a few other providers (TBD).

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging.providers import DatadogLogFormatter
    
logger = Logger(service="payment", logger_formatter=DatadogLogFormatter())
logger.info("hello")

Output

{
     "datetime": "2021-11-22 15:32:02,145", 
     "level": "INFO", 
     "message": "Request from Alice received",
     "requestId": "9b1deb4d-3b7d-4bad"
}

Custom logger usage

If the customer would like to use another observability provider, or define their own logger functions, we will define an interface that the customer can implement and pass in to the Logger class.

Out of scope

Sending logs from Powertools to the customer's desired observability platform will not be in the scope of this project. The implementation should only support modifying the output of the Logger so that the customer can push them to their platform of choice using the Lambda Handler.

Potential challenges

We need to determine which platforms we want to support out-of-the-box (apart from Datadog).

Dependencies and Integrations

We will have to integrate with (and thus, have a dependency on) Datadog and any other platforms we decide to support out-of-the-box.

Alternative solutions

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[heitorlessa]

reading!

[heitorlessa]

Thanks a lot @seshubaws for taking the time in creating this! I'm so excited we're kicking the observability provider discussion.

As I was reading, one thing struck me: only Python and TypeScript already provide this capability. This means we might not need an actual Provider capability but more of an improvement to the ergonomics for customers to bring their own output.

In Python, we use the standard logging library which is absurdly flexible yet why it's so complex to configure it. For Powertools Logger, customers tend to migrate from standard logging library or another loggers. Since we build atop of the standard logging library, we expose the same mechanisms for customers to override and bring their company standards, for example:

• You can bring your own Logging Handler to send logs anywhere you want: https://awslabs.github.io/aws-lambda-powertools-python/2.10.0/core/logger/#bring-your-own-handler
• You can bring your own Logging Formatter to control the output you want: https://awslabs.github.io/aws-lambda-powertools-python/2.10.0/core/logger/#bring-your-own-formatter

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging.formatter import LambdaPowertoolsFormatter


class CustomFormatter(LambdaPowertoolsFormatter):
    def serialize(self, log: dict) -> str:
        """Serialize final structured log dict to JSON str"""
        log["event"] = log.pop("message")  # rename message key to event
        return self.json_serializer(log)  # use configured json serializer


logger = Logger(service="payment", logger_formatter=CustomFormatter())
logger.info("hello")

In this case, logger_formatter would receive an instance of LambdaPowertoolsFormatter. Upon a new log that Logger already validated it has the correct logging levels, filters, fields like correlation ID, X-Ray ID, Lambda context and custom keys resolved, etc., it'd call serialize last and return. What I like about the logger_formatter is that it's explicit enough, customers can have additional logic in it, and we do the heavy lifting without them having to worry about logging levels, logging filters, and whatnot - if they do want the trouble, they can use BaseLambdaPowertoolsFormatter.

What I think we could do better here but didn't before is to either think of a simpler way since is already powerful enough, OR improve the existing mechanism by typing the log parameter with TypedDict for example.

What do you think?

---

Answering some comments and questions

Currently, the Logger automatically sends logs to AWS CloudWatch

Minor correction, Logger sends logs to standard output (stdout). That's why some customers even use it locally for scripts, or in platforms like Fargate/AppRunner/Beanstalk, despite not having access to all features. That's also true for all other languages. The reason we did this is to make it easier for customers to take control over stdout and transform/do as they please, with say sidecar containers... or in case of Lambda... Lambda Extensions Logs API.

We need to determine which platforms we want to support out-of-the-box (apart from Datadog).

We can ask the community. Turns out a customer shared that Logger output is recognized by Datadog out of the box when used with Logger(use_rfc3339=True). I had asked for a docs issue so make this more visible in a FAQ but customer lacked bandwidth (so did we :/).

An addendum to this proposal would be to have just one place users can specify which platform they'd like to use so they don't have to repeatedly configure the same observability provider setting for Logger, Metrics, and Tracer.

That's an interesting idea - would you mind expanding the RFC to cover an initial take? We heard some ideas in the past of having a central configuration.

Minor edits

• Added syntax highlighting to code snippets markdown (python)
• Corrected log output from Current logger experience
• Split code snippet from output to ease reading and comparing

[seshubaws]

Thank you for your comments @heitorlessa, it really clarifies a lot! I have revised the original RFC to integrate most of your comments, though I should call out some points:

1. What I think we could do better here but didn’t before is to either think of a simpler way since is already powerful enough, OR improve the existing mechanism by typing the log parameter with TypedDict for example.

In the revised RFC, I have proposed supporting Datadog and any other providers we want out of the box to make the current Logger Formatter simpler. We could add the TypedDict in addition to that, or do that as a fast follow, up to you.

2. We can ask the community. Turns out a customer shared that Logger output is recognized by Datadog out of the box when used with Logger(use_rfc3339=True). I had asked for a docs issue so make this more visible in a FAQ but customer lacked bandwidth (so did we :/).

I guess if we support Datadog formatting for logs out of the box like

logger = Logger(service="payment", logger_formatter=DatadogLogFormatter())

we can just do that behind the scenes.

Also, to determine the other providers we want to support out of the box, is this RFC the place where we are asking? Or is there some other forum for that kind of poll specifically?

3. "An addendum to this proposal would be to have just one place users can specify which platform they'd like to use so they don't have to repeatedly configure the same observability provider setting for Logger, Metrics, and Tracer."
   That's an interesting idea - would you mind expanding the RFC to cover an initial take? We heard some ideas in the past of having a central configuration.

Of course! Should we have that in a separate RFC though, since this one is just about the Logger and that would involve Logger, Tracer, and Metrics?

[heitorlessa]

awesome, I somehow missed your message @seshubaws (sorry!!). I'm finishing a documentation issue and going through all RFCs today

[heitorlessa]

Loved it! Gotta re-read the updated RFC now. As for your questions:

Also, to determine the other providers we want to support out of the box, is this RFC the place where we are asking? Or is there some other forum for that kind of poll specifically?

We can use the Poll feature in GitHub Discussions, then drive customers there from our docs, board, Discord, and social media :)

It'll also be good to find out which ones don't work outside the box (only DataDog didn't work out as far as I remember).

Of course! Should we have that in a separate RFC though, since this one is just about the Logger and that would involve Logger, Tracer, and Metrics?

Good idea! A new RFC would be better. I did learn from another customer interview last week that they tend to use one Logging provider (e.g., CloudWatch), and use another for Metrics and Traces primarily to optimize costs.

[heitorlessa]

That's it! That's exactly the simple developer experience customers love about Powertools - you're awesome @seshubaws!

Feel free to start the implementation (or let us know if you want us to!). Given how related it is, we could also address the TypedDict to make the authoring experience even better.

I've pinged a Serverless Partner SA for guidance on how to best ensure a fair chance for AWS Partners (whether to support a few out of the box, or do it based on customer demand and Partner's contribution). For now, assume Datadog as an example, as we can revert that easily if we need to.

Thanks