Feature request: Include Unwritten Log Entries On Error

[jasonwadsworth]

Description of the feature request

It would be great if the logger could collect some unwritten logs (maybe a configurable amount that defaults to 10 or something) and if there is an error logged it would also write any logs that were not already written.

Problem statement

I've always hated that the debug logs are never available when I need them. They are typically turned off in a production system, but when there is an error I'd like to have information from those logs. I don't want them most of the time, so really only when an error occurs. Sampling isn't a good option because the error may be intermittent and the logs will not be present for the relevant time.

Summary of the feature

Basically, at the point when the decision is made whether to log it would either log (current functionality) or store the log. If there was an error log it would log the error as well as everything stored (and flush the store).

Code examples

Benefits for you and the wider AWS community

Faster time to resolution on errors

Describe alternatives you've considered

I've looked at doing something like this myself but haven't tried it yet. As I mentioned, sampling doesn't accomplish what I'm looking for.

Additional context

Related issues, RFCs

[flochaz]

Thx for your suggestion ! Interesting idea that can definitely help reducing logging cost while having a verbose output on error. we will look into it .

[ijemmy]

I like this idea very much. It addresses my main use case for sampling rate. But it's more cost-efficient and we don't miss anything.

I suppose usage will be like this?

import { Logger } from '@aws-lambda-powertools/logger';
const logger = new Logger({
   logAllLevelsOnError: true,
});

In this case, the client needs to make sure that they catch any error and print out the log to get it flushed out.

It can be used with Middy's error-logger, to ensure that uncaught error are always handled.

[jasonwadsworth]

In this case, the client needs to make sure that they catch any error and print out the log to get it flushed out.

Yes, it is definitely dependent on the code catching and logging an error.

Came across this repo, which is doing what I'm talking about here. Doesn't seem too bad in this example, but I haven't looked at the logging code in this repo to see how complex it might be here.

[ijemmy]

There is a blog about this. Useful for implementation reference: https://dev.to/aws-builders/saving-on-aws-lambda-amazon-cloudwatch-logs-costs-51od

Notice the clear() function or the logs can leak to the next Lambda call.

[buggy]

I really like this idea.

Could I suggest using levelOnError instead of logAllLevelsOnError? This would provide additional flexibility by allowing the developer to set the log level on error instead of assuming they want everything.

Use case:

1. Set level to WARN and levelOnError to INFO.
2. Use WARN for warnings that should always be logged
3. Use INFO for additional information (like the event) that will allow debugging when an error occurs
4. Use DEBUG for low level debugging (i.e. logging each iteration of a loop)

Why?
Frequently I only need a little extra information, like the event, to be logged on error. By allowing the developer to set the log level on error I can use INFO for the stuff I want to see when something goes wrong but still have DEBUG littered through my code for when I need to do development / debugging. This is particularly helpful if you only keep the last N items to be written on error because I want some things I log (INFO) but not others (DEBUG).