Initialization of SDK client in `DynamoDBPersistenceStore` leads to an error when used with SnapStart under certain conditions

[roamingthings]

Yesterday one of my Lambda Function that is deployed using SnapStart ran into a problem that has been caused by
code that has been derived from this repository.

// AWS_LAMBDA_INITIALIZATION_TYPE has two values on-demand and snap-start
// when using snap-start mode, the env var creds provider isn't used and causes a fatal error if set
// fall back to the default provider chain if the mode is anything other than on-demand.
String initializationType = System.getenv().get(AWS_LAMBDA_INITIALIZATION_TYPE);
if (initializationType  != null && initializationType.equals(ON_DEMAND)) {
    ddbBuilder.credentialsProvider(EnvironmentVariableCredentialsProvider.create());
}

powertools-lambda-java/powertools-idempotency/src/main/java/software/amazon/lambda/powertools/idempotency/persistence/DynamoDBPersistenceStore.java

Line 100 in c185cb0

ddbBuilder.credentialsProvider(EnvironmentVariableCredentialsProvider.create());

An SDK client that has been initialized like that may fail to retrieve the credentials after an invocation has been timed out and the runtime is reused by subsequent invocations.

Also the error has been visible in my code, the code in this library may also be affected.

Expected Behavior

The client initialized inDynamoDBPersistenceStore will be able to retrieve its credentials after an previous invocation of the same runtime timed out.

Current Behavior

DynamoDBPersistenceStore fails when a previous invocation timed out.

Possible Solution

Always use the default credentials provider chain by removing the code shown above. This has also been the suggested solution provided by AWS support.

Steps to Reproduce (for bugs)

I've created a proof-of-concept project to reproduce the issue:
https://github.com/roamingthings/sdk-client-snapstart-error-poc

What probably happens is the following:

• the function is initialized when publishing a new version.
  • AWS_LAMBDA_INITIALIZATION_TYPE is snap-start
• the SDK client is configured to use the default credentials provider chain
• the snapshot is generated
• the function gets invoked
• the snapshot is resumed
• the function times out for any reason (not related to this issue)
• the function is invoked again
• the runtime re-instantiates the handler
  • • AWS_LAMBDA_INITIALIZATION_TYPE is on-demand
• the SDK client is also re-instantiated (it's not static in our case)
• the SDK client is configured to only use the environment variable credentials provider
• as soon as the SDK client is used it tries get the credentials and fails with `Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).

Environment

• Powertools for AWS Lambda (Java) version used: 1.16.0
• Packaging format (Layers, Maven/Gradle): Gradle, Dependency-Layer
• AWS Lambda function runtime:
• Debugging logs included in the proof-of-concept repository

[roamingthings]

I've created a proof-of-concept and updated the original issue.
https://github.com/roamingthings/sdk-client-snapstart-error-poc

[scottgerring]

Hey @roamingthings thanks for the detailed analysis and reproduction!
This relates to #1161

[scottgerring]

As a workaround anyone who hits this should be able to pass a client to the builder configured with the default credentials chain - here an example for idempotency/DynamoDB but the same pattern should be usable with the other modules that customize the credentials chain by default:

// Create a DDB client for use with the DynamoDbPersistenceStore builder
DynamoDbClient createClient() {
            return DynamoDbClient.builder()
                    .httpClientBuilder(UrlConnectionHttpClient.builder())
                    .region(Region.of(System.getenv(SdkSystemSetting.AWS_REGION.environmentVariable())))

powertools-lambda-java/powertools-idempotency/src/main/java/software/amazon/lambda/powertools/idempotency/persistence/DynamoDBPersistenceStore.java

Lines 415 to 418 in c185cb0

	public Builder withDynamoDbClient(DynamoDbClient dynamoDbClient) {
	this.dynamoDbClient = dynamoDbClient;
	return this;
	}