RFC: New environment variable POWERTOOLS_DEV to ease prototyping

[heitorlessa]

Is this related to an existing feature request or issue?

• Feature request: Simplified logger formatter when in AWS SAM Local powertools-lambda-python#1527
• Feature request: pretty printing logs in local development powertools-lambda-typescript#751

Which AWS Lambda Powertools utility does this relate to?

Logger, Event Handler, and potentially others

Summary

This RFC outlines the introduction of a new environment variable POWERTOOLS_DEV to ease local prototyping, debugging, and why not per utility env var.

Use case

Powertools optimizes certain outputs for costs (CloudWatch Logs) and to facilitate native integrations (CloudWatch Embedded Metric Format, Batch processing, etc.). It also disables certain behaviours such as tracing when running outside Lambda, so adding Powertools doesn't get in the way.

As our customer base increases, it diversifies its usage and thus lies the opportunity to adapt and improve. For example, customers are running their Lambda function locally in various forms: AWS Chalice, AWS SAM CLI, AWS Amplify CLI, Serverless framework, etc.

This increases our complexity as we try to keep track of potential sources of local emulation and their many environment variables. Notwithstanding customers who may also run Lambda Powertools in non-Lambda environments such as Glue jobs, Fargate, whether we provide support or not.

Proposal

Introduce POWERTOOLS_DEV boolean environment variable for the following effect:

Utility	Effect
Logger	Increase JSON indentation to 4. This will ease local debugging when running functions locally under emulators or direct calls while not affecting unit tests
Event Handler	Enable full traceback errors in the response, indented request/responses, and CORS in dev mode (*). This will deprecate POWERTOOLS_EVENT_HANDLER_DEBUG that achieves the same effect in Event Handler Debug Mode
Tracer	Disables tracing operations. Future-proof non-Lambda environments. As of now, we'll continue with auto-disable feature until we decide when (if) we should move beyond Lambda as a compute environment

This change future-proof additional convenience for customers and lower cognitive load on needing multiple env vars to the same end. We will create a new section in the documentation to keep track of behaviours this env var will enable.

Warning increased cost in log ingestion

We anticipate customers wanting to use POWERTOOLS_DEV in non-production AWS accounts. This might also occur unwillingly. Regardless of the intention, we should log a warning message to signal this feature is in use. This should help prevent an increase in log ingestion cost in production.

Out of scope

Change Tracer auto-disable behaviour in non-Lambda environments. This will cause a breaking change in customers unit tests, which is a great convenient Powertools feature.

We can revisit it when (if) we officially support compute beyond Lambda.

Potential challenges

An open question we have is: What if a customer intentionally deploys with POWERTOOLS_DEV=true? See Warning increased cost in log ingestion

Should we allow it or safely ignore it (e.g., POWERTOOLS_DEV && not TASK_ROOT)? Tracer only

Dependencies and Integrations

None

Alternative solutions

Introduce an environment variable per utility when the need arises, for example POWERTOOLS_LOGGER_INDENTED_LOGS, POWERTOOLS_EVENT_HANDLER_INDENTED_LOGS.

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript, and .NET

[benbridts]

Will this be expected to not be used when running (in a sandbox/dev) account on Lambda?

I might want to convenience of enabling all dev-functionality at once (logging the full event might also be one?), without it impacting how I see things in CloudWatch. POWERTOOLS_LOGGER_INDENTED_LOGS s a good example of things that I want different locally or when using CloudWatch

[heitorlessa]

Exactly, we won't impact the final runtime. I'll make that clearer in the RFC body.

I might want to convenience of enabling all dev-functionality at once (logging the full event might also be one?), without it impacting how I see things in CloudWatch

That's exactly the main argument to use POWERTOOLS_DEV locally instead of what was suggested in the original issue to change from JSON to non-JSON -- POWERTOOLS_DEV will only increase indentation so you can see it better locally but not change at Lambda runtime otherwise it'd increase costs.

[heitorlessa]

Updated the body of the RFC, and also posed a question if a customer accidentally deploys a function with POWERTOOLS_DEV=true instead of only using it locally -- lemme know if it's clearer.

---

Changes

NOTE: We should only change the experience locally, not when deployed at Runtime.

Potential challenges

A customer might accidentally deploys a Lambda function with POWERTOOLS_DEV=true, where this would change the logging output indentation from compact to pretty.

Should we have additional checks for it? e.g., POWERTOOLS_DEV && not TASK_ROOT

[benbridts]

I think POWERTOOLS_LOCAL would be a better name in that case.

I was thinking about it the other way. If this enables (in the future) things like more verbose logging, or logging of more things (inputs / outputs / ...), it might be nice to have it available in a Lambda Function too (if that function is in a developer's account and not getting a lot of traffic).

EDIT: or to put it more clearly: I assumed POWERTOOLS_DEV could also be used to toggle DEBUG-like functionality. I understand now that this is not the case

[heitorlessa]

POWERTOOLS_DEV could also be used for debugging if you intentionally deploy a function with it. We actually had issues with that in the past, where people would forget it and have a much higher bill in CloudWatch Logs, plus sensitive info being logged (or CORS in dev mode).

I do see where you're coming from tho - I made the wording crispier and brought in as a what if question before we settle on this.

In hindsight, I guess we can't protect customers 100% from forgetting that parameter, and putting a guard would prevent you from utilizing in non-prod env. I initially thought it'd be more useful in local executions, but I can see cases where you'd want that deployed when multiple functions are involved.

On the naming itself, the reason we didn't propose LOCAL was to not give the impression Powertools would provide a local execution mode of sorts (ala SAM CLI).

[benbridts]

I do see where you're coming from as you would have the intentions to have more verbose logging in non-prod environments, where this wouldn't work if we have a guard for it. I guess we can't protect customers 100% from forgetting that parameter, and putting a guard would prevent you from utilizing in non-prod env -- I'll improve the RFC.

If this will always switch indentation, I think it's fine to prevent me from using it (but maybe it should print an INFO one-liner to the logs). Setting POWERTOOLS_DEV to True and toggling indentation to be off, is the same effort as toggling individual debug settings.

On the naming itself, the reason we didn't propose LOCAL was to not give the impression Powertools would provide a local execution mode of sorts (ala SAM CLI)

Naming is always difficult :). _DEV gave me the impression that it was similar to DEBUG in tools like Django.

[heitorlessa]

Yeah, hmmm. Future wise, I can't see how we'd detect you're running on Fargate/Glue/Docker and auto-toggle that off at runtime.

What if we simply log a warning (warnings in Python) that you enabled POWERTOOLS_DEV so to instruct this might not be a good idea in prod?

That could solve our side on trying too hard to auto-toggle off this feature in scenarios we can't always predict, and keeping customers informed about the outcome when enabled.

[heitorlessa]

@dreamorosi @saragerion @rubenfonseca @leandrodamascena what do you think of POWERTOOLS_LOCAL name?

In hindsight, whatever name we settle, we could enable another one POWERTOOLS_DEBUG=true to automatically set logging as DEBUG for our internals - For Python, we ask customers to manually enable raw debugging similar to boto.

[heitorlessa]

From @jfuss

I like the idea of: “powertools_mode” and making the value an enum. Allows you to scale to other things in the future possibly.

My concern with true or false, it doesn’t leave you room if you need or want to add thing X but not y. But generally supportive of dev and believe should avoid “local” naming.

My only concern with an enum is that it opens the room for typos given this is an env var. We could technically have a default in the Enum (MODE.DEV)

[benbridts]

What if we simply log a warning (warnings in Python) that you enabled POWERTOOLS_DEV so to instruct this might not be a good idea in prod?

Would you log it everywhere, or only in lambda?

In the second case, I think the indentation would show it's not a great idea anyway.

In the first case, there is a balance between annoying and helpful, but warnings about security and/or cost seem to be on the helpful side (maybe it should log outside the handler, so that environments that keep the function warm don't log on every invocation)

[heitorlessa]

The former. Like we do in Metrics today - we will log a warning (anywhere) if you try to flush metrics without a metric per se - it might be intentional, but we won't raise an Exception unless you tell us to (raise_on_empty_metrics)

[benbridts]

The former

👍🏻

My only concern with an enum is that it opens the room for typos given this is an env var. We could technically have a default in the Enum (MODE.DEV)

If used as intended, that failure should be spotted fast ("I enabled better logging, and I'm not getting better logging"), adding a warning "DECV is not a valid value for POWERTOOLS_MODE" will also point in the right direction

However, I think as long as the intent is clear, a boolean is fine too. A ℹ️ in the docs saying that it will not be very useful when running with CloudWatch might be nice though.

[heitorlessa]

Hypothetically, say we have POWERTOOLS_MODE with three modes: DEV, PROD, DEBUG, where PROD is the default. Consider docs also outlines intent clearly.

If you use XPTO as a value, we log a warning with valid values, and fallback to PROD for safety - does that sound clear enough?