Feature request: Leverage new DynamoDB Failed Conditional Writes behavior

[brnkrygs]

Use case

The new idempotency utility uses conditional writes against DynamoDB for its locking mechanism.

When the conditional write fails, the record is considered locked and the request is already processed. In that case, a query is issued against DynamoDB to fetch the cached result and return it.

This means for this scenario that customers must wait and pay for both a write and read, even though the write fails.

On June 30, 2023, DynamoDB announced the ability for a failed conditional write to return a copy of the existing record. This would allow the package to skip performing the query, improving performance and lowering cost.

Solution/User Experience

In the idempotency utility, use the new DynamoDB conditional write behavior to return the current state of the item in the same conditional write operation. Remove the follow-up query.

Alternative solutions

No response

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[dreamorosi]

Hi @brnkrygs thank you for suggesting this improvement.

At least on the surface the proposal makes a lot of sense, before adding it to the backlog we'll have to investigate the feasibility based on the current request flow and understand if it fits as-is or we need rearchitecting.

We are going to start looking at this after we publish the first release.

[tolutheo]

Taking a look at this issue as discussed with @dreamorosi

[tolutheo]

Hi Andrea,

A couple of changes I am proposing to change in this feature request:

Implemented a custom error to contain existingRecord field here:

public constructor(message: string, existingRecord: IdempotencyRecord) {
    super(message);
    this.existingRecord = existingRecord;
  }

Error is then handled here using the code snippet below:

if (error instanceof ConditionalCheckFailedException) {
          if (!error.Item) {
            throw new Error('Item is undefined');
          }
          const Item = unmarshall(error.Item);
          throw new IdempotencyItemAlreadyExistsError(
            `Failed to put record for already existing idempotency key: ${record.idempotencyKey}`,
            new IdempotencyRecord({
              idempotencyKey: Item[this.keyAttr],
              status: Item[this.statusAttr],
              expiryTimestamp: Item[this.expiryAttr],
              inProgressExpiryTimestamp: Item[this.inProgressExpiryAttr],
              responseData: Item[this.dataAttr],
              payloadHash: Item[this.validationKeyAttr],
            })
          );
        }

To make use of the existingRecord field, it is referenced here:

const idempotencyRecord: IdempotencyRecord = e.existingRecord;

So the getRecord API call is not made again to fetch item that lead to failure of the conditional write, the code snippet below is removed from here:

const idempotencyRecord: IdempotencyRecord =
           await this.#persistenceStore.getRecord(
             this.#functionPayloadToBeHashed
           );

However the removal of that code causes a couple of unit tests listed below to fail with the error => The test expects getRecordSpy to be called once/twice/three times, but it is not called at all.:

1. IdempotencyHandler.test.ts
2. idempotencyDecorator.test.ts
3. makeHandlerIdempotent.test.ts
4. makeIdempotent.test.ts

To fix the issue we would need to remove the unit test getRecordSpy which is testing the getRecord code snippet mentioned above across those 4 unit test files.

For now I haven't removed the getRecord because the test files were failing and wanted to inform you about them first. Other changes made to unit test file here:

test('_putRecord throws Error when Item is undefined', async () => {
    // Prepare
    const persistenceLayer = new TestDynamoDBPersistenceLayer({
      tableName: dummyTableName,
    });
    const mockRecord = new IdempotencyRecord({
      idempotencyKey: 'test-key',
      status: 'INPROGRESS',
      expiryTimestamp: Date.now(),
    });

    DynamoDBClient.prototype.send = jest.fn().mockRejectedValueOnce(
      new ConditionalCheckFailedException({
        message: 'Conditional check failed',
        $metadata: {},
      })
    );
    await expect(persistenceLayer._putRecord(mockRecord)).rejects.toThrowError(
      'Item is undefined'
    );
  });