Failed to send data to X-Ray daemon. OSError: [Errno 90] Message too long

[ShengHow95]

I am trying to use aws-lambda-powertools in my lambda function and when I run my lambda function, it came out with this error but my lambda function still able to run and complete. I suppose version 1.17.0 has already included streaming_threshold=0 but I would still get the same error with Message too long.

What were you trying to accomplish?
I would like to know if there is anyway to fix this issue without disabling x-ray tracing in my lambda function. What configuration or workaround I can have?

Environment

• Powertools version used: 1.17.0
• AWS Lambda function runtime:: Python 3.7

Logs

[ERROR]	2021-06-11T16:37:43.245Z	e949c1fb-fb68-41c3-a77d-736cef47ec04	Failed to send entity to Daemon.
Traceback (most recent call last):
  File "/opt/python/aws_xray_sdk/core/emitters/udp_emitter.py", line 41, in send_entity
    self._send_data(message)
  File "/opt/python/aws_xray_sdk/core/emitters/udp_emitter.py", line 63, in _send_data
    self._socket.sendto(data.encode('utf-8'), (self._ip, self._port))
OSError: [Errno 90] Message too long

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[heitorlessa]

Hey @ShengHow95 thanks for raising it!

We indeed fixed it with that setting after X-Ray SDK fixed that piece -- I couldn't reproduce it :/

If you were to remove sensitive information, could you share the code you're using so we can try reproducing it?

If you could also share what event source you're using that would be great -- we originally saw this happening with SQS before the fix.

Thank you!

[heitorlessa]

I was thinking this over the weekend and one possibility is that the response returned by the handler or decorated function is bigger than 64K it reproduces this issue - that's because we automatically serialize and capture response and exceptions from decorated functions.

If that's the case, could you try disabling this feature to see if that solves it?

Docs: https://awslabs.github.io/aws-lambda-powertools-python/latest/core/tracer/#disabling-response-auto-capture

from aws_lambda_powertools import Tracer

@tracer.capture_method(capture_response=False)
def fetch_sensitive_information():
    return "sensitive_information"

@tracer.capture_lambda_handler(capture_response=False)
def handler(event, context):
    sensitive_information = fetch_sensitive_information()

Thank you!

[ShengHow95]

@heitorlessa, I have just tried to include capture_response=False in my code and it works. However, I would like to understand a little bit further about what would be the impact of having capture_response=False if I include them in all methods and lambda_handler? Will it stop the tracing for all my code?

[heitorlessa]

That's awesome to hear @ShengHow95!

Your code will continue to be traced with this flag. What changes is that we will not capture the entire response as tracing metadata from your handler or method when this is set.

I'm on mobile now, but if you open X-Ray, pick any trace ID, and you should see a Metadata tab -- capture_response=True (default) instructs Tracer to serialize the entire return of your function/method as metadata so you can troubleshoot that more easily, if you need.

However, if your handler and/or method response is larger than 64K than this error can happen.

You can still add tracing metadata anywhere in your code like parts of your response since it's big.

I'll create a new issue to update the docs this week to add this error under FAQ to help others who might bump this in the future.

[ShengHow95]

Alright cool. Thanks alot for your assistance @heitorlessa!