Bug: `makeFunctionIdempotent` wrapper requires payload property matching `dataKeywordArgument`

[brnkrygs]

Expected Behaviour

When I use the Function Wrapper pattern with the makeFunctionIdempotent function, the dataKeywordArgument should be used to determine which of the arguments of my wrapped function to use for the data payload logged in the idempotency persistence mechanism.

With this setup, I should be able to send a payload to my function with any JSON structure and have the idempotency mechanism run successfully.

Current Behaviour

When I send a payload that does not include a property in the root of the input that matches dataKeywordArgument, I get an error:

{
    "errorType": "Error",
    "errorMessage": "Failed to save record in progress. This error was  caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.",
    "cause": {
        "errorType": "TypeError",
        "errorMessage": "The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined",
        "code": "ERR_INVALID_ARG_TYPE",
        "stack": [
            "TypeError [ERR_INVALID_ARG_TYPE]: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined",
            "    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)",
            "    at new NodeError (node:internal/errors:399:5)",
            "    at Hash.update (node:internal/crypto/hash:109:11)",
            "    at lC.generateHash (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:204:10)",
            "    at lC.getHashedIdempotencyKey (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:259:49)",
            "    at lC.saveInProgress (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:132:28)",
            "    at io.processIdempotency (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:145:35)",
            "    at io.handle (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:115:27)",
            "    at u (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/makeFunctionIdempotent.ts:81:31)",
            "    at Runtime.lAe (/private/var/folders/4x/83k8wgzx2nj37bdnmj078nccln0f93/T/tmpg2h0cneq/app.ts:45:4)",
            "    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1086:29)"
        ]
    },
    "stack": [
        "Error: Failed to save record in progress. This error was  caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.",
        "    at io.processIdempotency (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:158:15)",
        "    at io.handle (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:115:16)"
    ]
}

Code snippet

Lambda handler function initialized by sam init using Hello World template, TypeScript

import { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';
import { IdempotencyConfig, makeFunctionIdempotent } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';

const IDEMPOTENCY_TABLE_NAME = process.env['IDEMPOTENCY_TABLE_NAME'] as string;
const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: IDEMPOTENCY_TABLE_NAME,
});

const processingFunction = async (payload: Record<string, APIGatewayProxyEvent>): Promise<APIGatewayProxyResult> => {
  // your code goes here here
  console.log('processingFunction(request).', payload);

  try {
    return {
      statusCode: 200,
      body: JSON.stringify({
        message: 'tire kick created',
      }),
    };
  } catch (err) {
    console.log(err);
    return {
      statusCode: 500,
      body: JSON.stringify({
        message: 'some error happened',
      }),
    };
  }
};

export const lambdaHandler = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
  console.log('have request in handler:', event);
  const result = makeFunctionIdempotent(processingFunction, {
    persistenceStore,
    dataKeywordArgument: 'payload',
  })(event);

  return result;
};

Invoke that works

Notice property that matches dataKeywordArgument in sample

sam remote invoke --event '{"payload":"abc124"}'

Invoke that fails

sam remote invoke --event '{"something_else":"thing"}'

Steps to Reproduce

1. In this environment: Latest build of powertools that includes the idempotency utility, Lambda runtime nodejs 18. No other dependencies. Set dataKeywordArgument to "payload". Lambda initialized with sam init using TypeScript Hello World template.
2. When I issue a sam remote invoke or use the Lambda test console to send a invoke payload that does not include a property named "payload", I get an Error.
3. When I issue a sam remote invoke or use the Lambda test console to send an invoke payload that includes a property named "payload" in the root of the JSON, it works.

Error: Failed to save record in progress. This error was caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.

Possible Solution

No response

Powertools for AWS Lambda (TypeScript) version

latest

AWS Lambda function runtime

18.x

Packaging format used

npm

Execution logs

For this log, `dataKeywordArgument` is set to `"payload"`

# Failure


INIT_START Runtime Version: nodejs:18.v8	Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:2755dc322c8dbb64760145d6403d14432af527bf4dd3cf03713aae10e0f8b552
START RequestId: e3e527fa-472a-4ee1-a709-699dda02ea77 Version: $LATEST
2023-07-03T17:43:16.560Z	e3e527fa-472a-4ee1-a709-699dda02ea77	INFO	have request in handler: {
  transaction_id: 'eee',
  resource: '/tire-kicks',
  path: '/tire-kicks',
  httpMethod: 'PUT',
  headers: {
    Accept: '*/*',
    'CloudFront-Forwarded-Proto': 'https',
    'CloudFront-Is-Desktop-Viewer': 'true',
    'CloudFront-Is-Mobile-Viewer': 'false',
    'CloudFront-Is-SmartTV-Viewer': 'false',
    'CloudFront-Is-Tablet-Viewer': 'false',
    'CloudFront-Viewer-ASN': '7922',
    'CloudFront-Viewer-Country': 'US',
    'content-type': 'application/x-www-form-urlencoded',
    Host: '98whvcov2m.execute-api.us-east-2.amazonaws.com',
    transaction_id: '123',
    'User-Agent': 'curl/7.88.1',
    Via: '2.0 e807928bf10df259d5d9aa45c69572c8.cloudfront.net (CloudFront)',
    'X-Amz-Cf-Id': 'Bly3yGg3y0hIdLl_cCWVcbZtahuyjNPZPA_zU_YliYCqw82IOhhQ_Q==',
    'X-Amzn-Trace-Id': 'Root=1-64a2ff98-210a43b91f2d5c8b0a2d3a3d',
    'X-Forwarded-For': '68.48.152.62, 15.158.47.105',
    'X-Forwarded-Port': '443',
    'X-Forwarded-Proto': 'https'
  },
  multiValueHeaders: {
    Accept: [ '*/*' ],
    'CloudFront-Forwarded-Proto': [ 'https' ],
    'CloudFront-Is-Desktop-Viewer': [ 'true' ],
    'CloudFront-Is-Mobile-Viewer': [ 'false' ],
    'CloudFront-Is-SmartTV-Viewer': [ 'false' ],
    'CloudFront-Is-Tablet-Viewer': [ 'false' ],
    'CloudFront-Viewer-ASN': [ '7922' ],
    'CloudFront-Viewer-Country': [ 'US' ],
    'content-type': [ 'application/x-www-form-urlencoded' ],
    Host: [ '98whvcov2m.execute-api.us-east-2.amazonaws.com' ],
    transaction_id: [ '123' ],
    'User-Agent': [ 'curl/7.88.1' ],
    Via: [
      '2.0 e807928bf10df259d5d9aa45c69572c8.cloudfront.net (CloudFront)'
    ],
    'X-Amz-Cf-Id': [ 'Bly3yGg3y0hIdLl_cCWVcbZtahuyjNPZPA_zU_YliYCqw82IOhhQ_Q==' ],
    'X-Amzn-Trace-Id': [ 'Root=1-64a2ff98-210a43b91f2d5c8b0a2d3a3d' ],
    'X-Forwarded-For': [ '68.48.152.62, 15.158.47.105' ],
    'X-Forwarded-Port': [ '443' ],
    'X-Forwarded-Proto': [ 'https' ]
  },
  queryStringParameters: null,
  multiValueQueryStringParameters: null,
  pathParameters: null,
  stageVariables: null,
  requestContext: {
    resourceId: 'ox2bhw',
    resourcePath: '/tire-kicks',
    httpMethod: 'PUT',
    extendedRequestId: 'Hfzf7H9ViYcF3jA=',
    requestTime: '03/Jul/2023:17:04:24 +0000',
    path: '/Prod/tire-kicks',
    accountId: '498170313562',
    protocol: 'HTTP/1.1',
    stage: 'Prod',
    domainPrefix: '98whvcov2m',
    requestTimeEpoch: 1688403864837,
    requestId: '53bd769a-b1e4-4a74-84fa-364555b34d11',
    identity: {
      cognitoIdentityPoolId: null,
      accountId: null,
      cognitoIdentityId: null,
      caller: null,
      sourceIp: '68.48.152.62',
      principalOrgId: null,
      accessKey: null,
      cognitoAuthenticationType: null,
      cognitoAuthenticationProvider: null,
      userArn: null,
      userAgent: 'curl/7.88.1',
      user: null
    },
    domainName: '98whvcov2m.execute-api.us-east-2.amazonaws.com',
    apiId: '98whvcov2m'
  },
  body: '{"transactionId":"10a"}',
  isBase64Encoded: false
}
2023-07-03T17:43:16.579Z	e3e527fa-472a-4ee1-a709-699dda02ea77	WARN	No value found for idempotency_key. jmespath: 
2023-07-03T17:43:22.159Z	e3e527fa-472a-4ee1-a709-699dda02ea77	ERROR	Invoke Error 	
{
    "errorType": "Error",
    "errorMessage": "Failed to save record in progress. This error was  caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.",
    "cause": {
        "errorType": "TypeError",
        "errorMessage": "The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined",
        "code": "ERR_INVALID_ARG_TYPE",
        "stack": [
            "TypeError [ERR_INVALID_ARG_TYPE]: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined",
            "    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)",
            "    at new NodeError (node:internal/errors:399:5)",
            "    at Hash.update (node:internal/crypto/hash:109:11)",
            "    at lC.generateHash (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:204:10)",
            "    at lC.getHashedIdempotencyKey (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:259:49)",
            "    at lC.saveInProgress (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/persistence/BasePersistenceLayer.ts:132:28)",
            "    at io.processIdempotency (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:145:35)",
            "    at io.handle (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:115:27)",
            "    at u (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/makeFunctionIdempotent.ts:81:31)",
            "    at Runtime.lAe (/private/var/folders/4x/83k8wgzx2nj37bdnmj078nccln0f93/T/tmpg2h0cneq/app.ts:45:4)",
            "    at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1086:29)"
        ]
    },
    "stack": [
        "Error: Failed to save record in progress. This error was  caused by: The \"data\" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined.",
        "    at io.processIdempotency (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:158:15)",
        "    at io.handle (/deps/f68a7c30-4824-4150-bbd4-4456da07f1f4/node_modules/@aws-lambda-powertools/idempotency/src/IdempotencyHandler.ts:115:16)"
    ]
}

END RequestId: e3e527fa-472a-4ee1-a709-699dda02ea77

Success

REPORT RequestId: e3e527fa-472a-4ee1-a709-699dda02ea77	Duration: 5664.42 ms	Billed Duration: 5665 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	Init Duration: 395.56 ms	
XRAY TraceId: 1-64a308b4-0e12a3155fe94234496bc4df	SegmentId: 5339526d7ed02274	Sampled: true	
START RequestId: fe90db53-cee2-4d46-ae9e-2f8eaea8d690 Version: $LATEST
2023-07-03T17:43:38.767Z	fe90db53-cee2-4d46-ae9e-2f8eaea8d690	INFO	have request in handler: {
  payload: 'eee',
  resource: '/tire-kicks',
  path: '/tire-kicks',
  httpMethod: 'PUT',
  headers: {
    Accept: '*/*',
    'CloudFront-Forwarded-Proto': 'https',
    'CloudFront-Is-Desktop-Viewer': 'true',
    'CloudFront-Is-Mobile-Viewer': 'false',
    'CloudFront-Is-SmartTV-Viewer': 'false',
    'CloudFront-Is-Tablet-Viewer': 'false',
    'CloudFront-Viewer-ASN': '7922',
    'CloudFront-Viewer-Country': 'US',
    'content-type': 'application/x-www-form-urlencoded',
    Host: '98whvcov2m.execute-api.us-east-2.amazonaws.com',
    transaction_id: '123',
    'User-Agent': 'curl/7.88.1',
    Via: '2.0 e807928bf10df259d5d9aa45c69572c8.cloudfront.net (CloudFront)',
    'X-Amz-Cf-Id': 'Bly3yGg3y0hIdLl_cCWVcbZtahuyjNPZPA_zU_YliYCqw82IOhhQ_Q==',
    'X-Amzn-Trace-Id': 'Root=1-64a2ff98-210a43b91f2d5c8b0a2d3a3d',
    'X-Forwarded-For': '68.48.152.62, 15.158.47.105',
    'X-Forwarded-Port': '443',
    'X-Forwarded-Proto': 'https'
  },
  multiValueHeaders: {
    Accept: [ '*/*' ],
    'CloudFront-Forwarded-Proto': [ 'https' ],
    'CloudFront-Is-Desktop-Viewer': [ 'true' ],
    'CloudFront-Is-Mobile-Viewer': [ 'false' ],
    'CloudFront-Is-SmartTV-Viewer': [ 'false' ],
    'CloudFront-Is-Tablet-Viewer': [ 'false' ],
    'CloudFront-Viewer-ASN': [ '7922' ],
    'CloudFront-Viewer-Country': [ 'US' ],
    'content-type': [ 'application/x-www-form-urlencoded' ],
    Host: [ '98whvcov2m.execute-api.us-east-2.amazonaws.com' ],
    transaction_id: [ '123' ],
    'User-Agent': [ 'curl/7.88.1' ],
    Via: [
      '2.0 e807928bf10df259d5d9aa45c69572c8.cloudfront.net (CloudFront)'
    ],
    'X-Amz-Cf-Id': [ 'Bly3yGg3y0hIdLl_cCWVcbZtahuyjNPZPA_zU_YliYCqw82IOhhQ_Q==' ],
    'X-Amzn-Trace-Id': [ 'Root=1-64a2ff98-210a43b91f2d5c8b0a2d3a3d' ],
    'X-Forwarded-For': [ '68.48.152.62, 15.158.47.105' ],
    'X-Forwarded-Port': [ '443' ],
    'X-Forwarded-Proto': [ 'https' ]
  },
  queryStringParameters: null,
  multiValueQueryStringParameters: null,
  pathParameters: null,
  stageVariables: null,
  requestContext: {
    resourceId: 'ox2bhw',
    resourcePath: '/tire-kicks',
    httpMethod: 'PUT',
    extendedRequestId: 'Hfzf7H9ViYcF3jA=',
    requestTime: '03/Jul/2023:17:04:24 +0000',
    path: '/Prod/tire-kicks',
    accountId: '498170313562',
    protocol: 'HTTP/1.1',
    stage: 'Prod',
    domainPrefix: '98whvcov2m',
    requestTimeEpoch: 1688403864837,
    requestId: '53bd769a-b1e4-4a74-84fa-364555b34d11',
    identity: {
      cognitoIdentityPoolId: null,
      accountId: null,
      cognitoIdentityId: null,
      caller: null,
      sourceIp: '68.48.152.62',
      principalOrgId: null,
      accessKey: null,
      cognitoAuthenticationType: null,
      cognitoAuthenticationProvider: null,
      userArn: null,
      userAgent: 'curl/7.88.1',
      user: null
    },
    domainName: '98whvcov2m.execute-api.us-east-2.amazonaws.com',
    apiId: '98whvcov2m'
  },
  body: '{"transactionId":"10a"}',
  isBase64Encoded: false
}
2023-07-03T17:43:38.768Z	fe90db53-cee2-4d46-ae9e-2f8eaea8d690	WARN	Could not determine remaining time left. Did you call registerLambdaContext on IdempotencyConfig?
END RequestId: fe90db53-cee2-4d46-ae9e-2f8eaea8d690
REPORT RequestId: fe90db53-cee2-4d46-ae9e-2f8eaea8d690	Duration: 712.45 ms	Billed Duration: 713 ms	Memory Size: 128 MB	Max Memory Used: 128 MB	
XRAY TraceId: 1-64a308ca-2c283e332802034532f10fed	SegmentId: 332106327066e687	Sampled: true	

[dreamorosi]

Hi @brnkrygs thank you for reporting this, we'll be taking a closer look and get back to you here.

[dreamorosi]

Hi Brian, thank you again for reporting this.

As a result of this feedback, as well as the others you have shared, we have decided to change the API and remove the dataKeywordArgument entirely, in favor of a dataIndexArgument one.

The full detailed explanation of why we made this change can be found in the PR body of #1579 (section titled dataKeywordArgument vs dataIndexArgument).

At a high level, we realized that dataKeywordArgument was confusing and actually detrimental in the sense that it was duplicating some of the function that should have been covered by eventKeyJmespath.

The main function of dataKeywordArgument should have been to tell the utility which argument of a function should be used for the idempotency, as opposed to what subset/portion of the argument (something covered by eventKeyJmespath).

With this in mind, and considering that JavaScript doesn't have a notion of keyword arguments, we have decided to replace the option with a more aptly named dataIndexArgument one. In JS the only way to identify arguments is by index (read here to understand why), so in order to allow customers to tell the utility which argument to use, they now have to specify the index of that argument. If no argument is specified, we use the first one.

This can be used in conjunction with the eventKeyJmespath option, to both tell the utility which one and what part/subset of an argument should be used for idempotency. The new README for the utility (merged in the next few hours) will include examples for all those cases.

Let me know if this helps, it'd be amazing if you could try the same implementation with the new logic and let me know if it fixes the issue (I can provide a build if needed, just ping me).

[brnkrygs]

Thanks for your effort on this @dreamorosi, I would love to take another run at the new implementation. I have a set of tests I want to run against it 😄

Will ping you about a build.

[brnkrygs]

The new build is looking good! No more exceptions about the data argument.