Auto-disable not happening when running outside AWS Lambda environment

[bepetersn]

Expected Behaviour

This is a screenshot from your tracer documentation:

My expectation would be that, if I write a lambda that includes this at the module level...

from aws_lambda_powertools import Tracer
tracer = Tracer()
...

and employ a requirements.txt like this...

aws-lambda-powertools==2.22.0
aws-xray-sdk==2.12.0 

... and then, execute my lambda code locally using SAM, eg. sam local invoke, then powertools' tracer turns off, including of course not making warning logs related to tracer functionality (among other aspects, e.g. actually not PUTing traces as well).

Current Behaviour

When I execute my lambda via sam local invoke

[WARNING] 2024-01-17T18:04:45.015Z Subsegment ssm discarded due to Lambda worker still initializing
[WARNING] 2024-01-17T18:04:45.387Z No subsegment to end.
[WARNING] 2024-01-17T18:04:45.391Z e0e04135-5a6b-4be6-ba55-0051fd794bee Subsegment ## lambda_handler discarded due to Lambda worker still initializing
...
[WARNING] 2024-01-17T18:04:45.392Z e0e04135-5a6b-4be6-ba55-0051fd794bee Subsegment ## app.process_trip discarded due to Lambda worker still initializing
...
[WARNING] 2024-01-17T18:04:45.392Z e0e04135-5a6b-4be6-ba55-0051fd794bee Subsegment dynamodb discarded due to Lambda worker still initializing
[WARNING] 2024-01-17T18:04:45.611Z e0e04135-5a6b-4be6-ba55-0051fd794bee No subsegment to end.

Code snippet

Some of the relevant code:

from aws_lambda_powertools import Tracer
from aws_lambda_powertools.logging import correlation_paths
... 

@logger.inject_lambda_context(correlation_id_path=correlation_paths.LAMBDA_FUNCTION_URL)
@tracer.capture_lambda_handler
def lambda_handler(event, context):
    ...

@tracer.capture_method
def process_trip(webhook: Webhook, qa: Optional[str]) -> None:
    ...

I am also using AWS's SSM and DynamoDB services via boto3 in the "cold start" section of my lambda.

Possible Solution

I can get these warnings to go away by specifying an environment variable, as described here: #1898 (POWERTOOLS_TRACE_DISABLED), however this doesn't satisfy the documentation's claim that tracer is auto-disabled when not running in an AWS Lambda environment.

I am given to understanding that SAM's local invoke command does not create traces/segments/sub-segments out of the box even when tracing is turned on--nor is there an X-Ray daemon to collect / PUT those traces. Can you confirm if I'm wrong in that? That would partially put my issue outside the scope of this project. For example, the log Subsegment dynamodb discarded due to Lambda worker still initializing isn't obviously being generated by the powertools library--although maybe it is. You all would know better.

However, I still see logs that look like below. I am very confident this is coming from Powertools:

Subsegment ## lambda_handler discarded due to Lambda worker still initializing

Is the documentation incorrect? Is the environment variable required?

Steps to Reproduce

I don't have a minimal reproduction case. If I really need that, I can put it together. Want to understand first if I'm simply confused, as I am confident this will have already come up many times by now.

Powertools for AWS Lambda (Python) version

2.22.0

AWS Lambda function runtime

3.11

Packaging format used

PyPi

Debugging logs

No response

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Hey @bepetersn! Thanks for bringing up this question! Let me start by saying that the reason we disable Tracer outside of the Lambda environment is that we rely on the Root Segment created by Lambda and then we can create subsegments from it. We may need to investigate how to allow customers to use our Tracer utility outside of the Lambda environment, but this is the behavior for now.

Just out of curiosity and before I continue the explanation, how is this impacting your experience or environment? I'm interested in hearing this and seeing if we can think of any solutions or something else.

Possible Solution

I can get these warnings to go away by specifying an environment variable, as described here: #1898 (POWERTOOLS_TRACE_DISABLED), however this doesn't satisfy the documentation's claim that tracer is auto-disabled when not running in an AWS Lambda environment.

The reason this is happening is that we verify if it is a Lambda environment by checking the LAMBDA_TASK_ROOT env variable and AWS SAM also sets this variable. I don't have the information of whether this was intentional or not, not disabling Tracer in AWS SAM and I need to talk to my peers. However, it seems to make sense to disable it too, since we don't have the root segment in AWS SAM and X-Ray SDK emits these warnings.

For example, the log Subsegment dynamodb discarded due to Lambda worker still initializing isn't obviously being generated by the powertools library--although maybe it is. You all would know better.

However, I still see logs that look like below. I am very confident this is coming from Powertools:

Subsegment ## lambda_handler discarded due to Lambda worker still initializing

This is generated by X-Ray SDK when Powertools try to create a subsegment.

Is the documentation incorrect? Is the environment variable required?

With that explained, it would be interesting to hear from other maintainers and the community if we need to verify that it is running in a SAM environment and disable the Tracer utility, or update our documentation to add SAM as an exception. I'm adding labels to classify this issue.

Thanks.

[leandrodamascena]

Hi @bepetersn! Thanks for your patience and helping us to make Powertools for AWS Lambda even better! I have spoken to my colleagues and we will revert to this PR that removed the SAM CLI environment check.

Additionally, Powertools for AWS Lambda (TypeScript) has an additional check to validate that it is an Amplify environment too. We can add it in the same way.

Do you want to send a PR to add this? We'd love to have your first contribution here.