combination of managed layer AWSSDKPandas-Python312 and AWSLambdaPowertoolsPythonV2 result in ModuleNotFoundError: No module named '_cffi_backend'

[Eerk1975]

Describe the bug

For my Python 3.12 Lambda function I need to combine Powertools and SDK for Pandas. I would like to use the managed layers for this.
I added layers: arn:aws:lambda:eu-west-1:336392948345:layer:AWSSDKPandas-Python312:4 and arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:64

this results in:

Function Logs
rom cryptography.hazmat.primitives.hashes import HashAlgorithm
File "/opt/python/cryptography/hazmat/primitives/hashes.py", line 9, in
from cryptography.hazmat.bindings._rust import openssl as rust_openssl
pyo3_runtime.PanicException: Python API call failed
INIT_REPORT Init Duration: 4103.15 ms Phase: init Status: error Error Type: Runtime.ExitError
ModuleNotFoundError: No module named '_cffi_backend'
thread '' panicked at /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.20.2/src/err/mod.rs:788:5:
Python API call failed
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
Traceback (most recent call last):
File "/var/runtime/bootstrap.py", line 63, in
main()
File "/var/runtime/bootstrap.py", line 60, in main
awslambdaricmain.main([os.environ["LAMBDA_TASK_ROOT"], os.environ["_HANDLER"]])
File "/var/lang/lib/python3.12/site-packages/awslambdaric/main.py", line 21, in main
bootstrap.run(app_root, handler, lambda_runtime_api_addr)
File "/var/lang/lib/python3.12/site-packages/awslambdaric/bootstrap.py", line 478, in run
request_handler = _get_handler(handler)
^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.12/site-packages/awslambdaric/bootstrap.py", line 53, in _get_handler
m = importlib.import_module(modname.replace("/", "."))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 994, in exec_module
File "", line 488, in _call_with_frames_removed
File "/var/task/lambda_function.py", line 2, in
import awswrangler as wr
File "/opt/python/awswrangler/init.py", line 10, in
from awswrangler import (
File "/opt/python/awswrangler/mysql.py", line 27, in
pymysql = _utils.import_optional_dependency("pymysql")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/awswrangler/_utils.py", line 147, in import_optional_dependency
module = importlib.import_module(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/pymysql/init.py", line 78, in
from . import connections # noqa: E402
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/pymysql/connections.py", line 13, in
from . import _auth
File "/opt/python/pymysql/_auth.py", line 9, in
from cryptography.hazmat.primitives import serialization, hashes
File "/opt/python/cryptography/hazmat/primitives/serialization/init.py", line 7, in
from cryptography.hazmat.primitives._serialization import (
File "/opt/python/cryptography/hazmat/primitives/_serialization.py", line 10, in
from cryptography.hazmat.primitives.hashes import HashAlgorithm
File "/opt/python/cryptography/hazmat/primitives/hashes.py", line 9, in
from cryptography.hazmat.bindings._rust import openssl as rust_openssl
pyo3_runtime.PanicException: Python API call failed
INIT_REPORT Init Duration: 58089.99 ms Phase: invoke Status: error Error Type: Runtime.ExitError
START RequestId: 5d231810-7e77-403c-8727-30dba414f36f Version: $LATEST
RequestId: 5d231810-7e77-403c-8727-30dba414f36f Error: Runtime exited with error: exit status 1

I also tried the same with Python 3.11 and corresponding managed layers, but tis resulted in the same behaviour.
removing one of the layers resulted in behaviour I expected.

How to Reproduce

as code in the lambda function I only added (as test):

import awswrangler as wr

def lambda_handler(event, context):

    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }``` 



### Expected behavior

Hello from Lambda!'

### Your project

_No response_

### Screenshots

_No response_

### OS

Lambda

### Python version

3.12

### AWS SDK for pandas version

3.6.0

### Additional context

_No response_

[Eerk1975]

an extra test I did: the combination of arn:aws:lambda:eu-west-1:336392948345:layer:AWSSDKPandas-Python311:6 and arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:11 in Python 3.11 did work

[jaidisido]

So here's what is going on here. awswrangler packages pymysql as a dependency in its lambda layers. In turn, pymysql has an optional dependency on the cryptography package. Because it's optional, we don't package it in the zip layer and therefore pymysql does not try to import it. This explains why you have no issue when using the awswrangler layer on its own.

On the other hand, recent versions of aws_lambda_powertools do package cryptography as a dependency in their layer. As a result, when both layers are used, pymysql detects that cryptography is available from powertools and it tries to import it.

Now on the exception No module named '_cffi_backend'. There seems to be an issue with the way cryptography was packaged in the powertools layer. I am able to reproduce the exception with this simple code:

import json

from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import (
    AWSEncryptionSDKProvider,
)

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

which also raises No module named _cffi_backend and as you can see does not involve awswrangler. From the little research I have done, it looks like powertools uses the same base image to package their layers but cffi is sensitive to the base python version on which it's installed:
https://stackoverflow.com/questions/76443834/aws-lambda-python-3-10-no-module-named-cffi-backend
We have the same issue with numpy and this is why we have one layer per python version when powertools does not.

In short, this is a powertools specific issue that they need to solve. I can open an issue with them to inform them.

[heitorlessa]

Powertools for AWS Lambda on-call checking in - we're investigating. Thank you so much for the preliminary investigation.

[heitorlessa]

we're making a patch release now to remove aws-encryption-sdk from the Powertools Layer. I'll update as soon as it's out -- it'll be version 65.

[jaidisido]

After testing with version 2.34.2 of powertools using layers:

• arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:65
• arn:aws:lambda:eu-west-1:336392948345:layer:AWSSDKPandas-Python312:4

on Python 3.12, I can confirm that the issue is fixed, thanks @heitorlessa. Closing the ticket

[heitorlessa]

an extra note for whoever bumps into this -- we will be investigating the operational effort in adding a new layer per Python version in the near future.