Feature request: allow custom formatter to `formatAttributes` on all attributes (not only the base attributes)

[ghdoergeloh]

Use case

In my use case there is a compliance that enforces the developer to put some basic attributes in the root level of a log json. All the other additional data like a error or other data to identify the reason or context of that log should be placed under a attribute "data".

Since the formatAttributes function of a Formatter only gets the unformattedBaseAttributes it is not possible to sort or format the other attributes.

Solution/User Experience

I would like to pass any attribute to the logs and enforce a nice ordering of the log in the formatter:

export class MyLogFormatter extends LogFormatter {
  public formatAttributes(attributes: UnformattedAttributes): LogAttributes {
    const {
      environment,
      serviceName,
      sampleRateValue,
      lambdaContext,
      xRayTraceId,
      awsRegion,
      logLevel,
      timestamp,
      message,
      ...data
    } = attributes;
    return {
      id: randomUUID(),
      timestamp: timestamp.getTime(),
      logLevel: logLevel,
      serviceName: serviceName,
      message: message,
      xrayTraceId: xRayTraceId,
      requestId: lambdaContext?.awsRequestId,
      data,
    };
  }
}

export const logger = new Logger({
  logFormatter: new MyLogFormatter(),
});

logger.error('This is my message', { error: new Error('something bad happened'), contextInfo: 1234 })

which should then result in a object like this:

  {
      "id": "...",
      "timestamp": "...",
      "logLevel": "error",
      "serviceName": "...",
      "message": "This is my message",
      "xrayTraceId": "...",
      "requestId": "...",
      "data": {
         "error": { "name":"...", "message": "..."},
         "contextInfo": 1234
    }

Alternative solutions

My current workaround is to extend the Logger Class and do the sorting in the debug,info,warn and error method:

interface LogAttributesWithMessageAndData extends LogAttributesWithMessage {
  data?: LogAttributes;
}

export class MyLogger extends Logger {
  public override debug(input: LogItemMessage, ...extraInput: LogItemExtraInput): void {
    super.debug(this.convertLogAttributes(input, extraInput));
  }

  public override info(input: LogItemMessage, ...extraInput: LogItemExtraInput): void {
    super.info(this.convertLogAttributes(input, extraInput));
  }

  public override warn(input: LogItemMessage, ...extraInput: LogItemExtraInput): void {
    super.warn(this.convertLogAttributes(input, extraInput));
  }

  public override error(input: LogItemMessage, ...extraInput: LogItemExtraInput): void {
    super.error(this.convertLogAttributes(input, extraInput));
  }


  private convertLogAttributes(input: LogItemMessage, extraInput: LogItemExtraInput): LogAttributesWithMessage {
    const attributes: LogAttributesWithMessageAndData = {
      message: '',
      data: {},
    };
    if (typeof input === 'string') {
      attributes.message = input;
    } else {
      const { message, ...data } = input;
      attributes.message = message;
      attributes.data = data;
    }
    extraInput.forEach((item) => {
      const dataAttributes: LogAttributes =
        item instanceof Error ? { error: item } : typeof item === 'string' ? { extra: item } : item;
      merge(attributes.data, dataAttributes);
    });

    if (attributes.data && !Object.keys(attributes.data).length) {
      delete attributes.data;
    }

    return attributes;
  }
}

export const logger = new MyLogger({
  logFormatter: new MyLogFormatter(),
});

logger.error('This is my message', { error: new Error('something bad happened'), contextInfo: 1234 })

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Python, Java

[saragerion]

Hi @ghdoergeloh, thanks for opening this issue.
I agree that this will be an improvement of the way the current logger works. We'll look into this possibility and get back to you with a more detailed perspective.

[ghdoergeloh]

Thank you.

I also found out, that I need to pass the LogFormatter to every child. Wouldn't it be nice to forward the parents LogFormatter as default to a child logger (via createChild).

So if logger.createChild({}) is called without logFormatter it could just get the same instance, the parent is using.

Should I create another issue / feature request for this?

[dreamorosi]

Yes, it'd be great if you could create a separate one, I see these two feature requests as separate units of work independent from each others.

[ghdoergeloh]

Done: #1264

[dreamorosi]

Thank you for opening this feature request.

One of the key value propositions of the Logger utility is to allow customers to bring their own formatter so that they can comply with their unique requirements, so I definitely see value in your proposal.

However I'd like to reframe/rephrase the feature request slightly.

Context

At the moment, the logic in charge of formatting attributes resides within the private method called createAndPopulateLogItem, which as you noticed, is called before a log is emitted.

By default, Logger comes with an opinionated LogFormatter, however customers can provide a custom one, which is expected to implement the formatAttributes method. This method is currently called only on the standard attributes and context-related ones if present.

As you pointed out, any attribute present in the extraInput is excluded from the formatting. This is true also for any other persistent attributes added to the logger.

The method was designed as such because the formatting done in the PowertoolLogFormatter is largely manual/hardcoded and aimed purely at standardizing the names of certain fields that are always guaranteed to exist and have stable names. All other fields (extraInput and persistentAttributes) are user-provided at runtime and as such we have decided to leave them as they are provided.

Proposed rephrasing/refocus

With this in mind, I think that the path forward should be to consider adding to custom log formatters, aka log formatters that extend the LogFormatter class and that are provided to the Logger at instantiation time via the logFormatter option, the ability to format also attributes provided as extraInput and persistentAttributes.

Proposed solutions / DX

Note
Ideally I would like to address this in a backward-compatible way and avoid a breaking change. However, I'm not sure this is possible so let's not limit ourselves and add this to the already ongoing discussion for Powertools v2.

The formatAttributes method currently has the following signature:

formatAttributes(attributes: UnformattedAttributes): LogAttributes

As discussed for now we are only passing some attributes to it, and excluding any extraInput and persistentAttributes.

Option 1

To give full control to a custom log formatter, all three types of attributes should be passed to it, which would require to change the method signature to something like:

formatAttributes(attributes: UnformattedAttributes, persistentLogAttributes: LogAttributes, extraInput: LogItemExtraInput): LogAttributes

This would allow us to move some of the some of the logic currently in the createAndPopulateLogItem method into PowertoolLogger, as well as potentially simplify the merging of these attributes.

At the same time, and more interestingly, this would give full control to users extending the LogFormatter class to truly apply any formatting and any sorting they like.

This is how Powertools for Python currently behaves, however changing out library's logger now would represent a breaking change and will require us to wait for a major version release.

Option 2

A non-breaking way of allowing some type of formatting, but not sorting would be to extend theLogFormatter interface and add two methods like:

formatPersistentAttributes(persistentLogAttributes: LogAttributes): void

formatExtraAttributes(extraAttributes: LogAttributes): void

which could then optionally overloaded by customers wanting to format one or the other group of attributes. These two methods would be called as part of the createAndPopulateLogItem chain.

While this could be a decent stop-gap solution, I'm not convinced it's the best way forward for the long term and in line with the key value proposition of Logger as an utility - which is to provide full customization of the log format.

---

What do you think? Would appreciate people's opinion on this + additional / alternative proposals.