Feature request: TypeScript: Log Metrics with JSON (not console.log)

[cortexcompiler]

Use case

We would like our Metrics to be logged in pure JSON to match the EMF document structure for addMetric() and the logMetrics() middleware. This will allow the logs to be machine readable and ingested by aggregators. Instead we are seeing the EMF JSON prefixed with the timestamp, request Id, and log level, because the code uses console.log()

We use the Powertools for AWS Lambda (TypeScript) Metrics and Powertools for AWS Lambda (TypeScript) Logger. The Logger works fine because it uses its own console instance.

This is an example of one of the log events from CloudWatch:

2023-10-25T17:38:05.273Z	274cced9-0439-4a76-befe-eeef0ec34460	INFO	
{
    "_aws": {
        "Timestamp": 1698255485273,
        "CloudWatchMetrics": [
            {
                "Namespace": "xxxxxx",
                "Dimensions": [
                    [
                        "service",
                        "environment",
                        "function_name"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "ColdStart",
                        "Unit": "Count"
                    }
                ]
            }
        ]
    },
    "service": "xxxxxx",
    "environment": "test",
    "function_name": "xxxxxx",
    "ColdStart": 1
}

Solution/User Experience

I would like to see the metrics logged with just JSON similar to how the logs are output from the Powertools Logger package.

• Option 1: Stop using standard console.log() and do something similar to the Powertools Logger, or even re-use it.
  • Con: This would not technically be backwards compatible and could in theory break folks if they are parsing their logs in a specific way
  • Pro: Everyone now gets easily machine readable metrics logs
• Option 2: Update addMetric() and logMetrics() to accept an option indicating the desire for pure JSON output.
  • Pro: Can be done in a backwards compatible manner
  • Con: requires code changes for folks that want JSON logging

Alternative solutions

An alternative would be for us to stop using PowerTools metrics and EMF and use our aggregator's metrics, or convert to OpenTelemetry to be vendor agnostic.

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[dreamorosi]

Hi @cortexcompiler thank you so much for taking the time to file this feature request.

As you mentioned at the moment the Metrics utility uses console object in the Node.js Lambda runtime, which automatically appends these fields to the content. For the Logger utility we received an early feature request (#747) that asked us to remove these fields for the same exact reason.

I can see the value of adopting a similar strategy for the Metrics utility and between the two options you propose - at least for now - I'd be more inclined to go with option 1. The main reasons for this choice are keeping a relatively simple interface, and the fact that we are currently working on our next major release (#1714) so there's no better time than now to make this type of correction.

Before making a final decision I'd like to take a bit of time to: 1/ test whether or not CloudWatch is still able to parse the EMF from the logs even after removing timestamp & request id from the log, and 2/ understand whether this has any impact for an eventual integration with other 3rd party observability providers.

[dreamorosi]

Hi @cortexcompiler, after looking into this I don't see any issue with supporting this.

I'm working on a PR and the change will likely be released in the next version.