Skip to content

Bug: payload validation is skipped when item is returned via ReturnValuesOnConditionCheckFailure #3781

Closed
@dreamorosi

Description

@dreamorosi

Expected Behaviour

When making a function idempotent via the @idempotent decorator and enabling payload validation the Idempotency utility should raise a payload validation exception when the payload in subsequent requests differs from the one stored.

With this configuration:

persistence_layer = DynamoDBPersistenceLayer(table_name=os.environ["TABLE_NAME"])
config = IdempotencyConfig(
    event_key_jmespath='["rcn", "subscriptionType"]',
    payload_validation_jmespath="customerAccounts",
)

we are using two fields from the payload to generate the idempotency hash (rcn and subscriptionType) but we are using a different key (customerAccounts) for payload validation.

If we send the following two requests:

{
  "rcn": "RCN-000-000-000-000",
  "customerAccounts": {
    "customerAccount": [
      {
        "accountId": "12345",
        "accountType": "someaccounttype"
      }
    ]
  },
  "subscriptionType": "somesubscriptiontype"
}

and

{
  "rcn": "RCN-000-000-000-000",
  "customerAccounts": {
    "customerAccount": [
      {
        "accountId": "54321",
        "accountType": "someaccounttype"
      }
    ]
  },
  "subscriptionType": "somesubscriptiontype"
}

These two requests should be considered idempotent because the compound value of rcn and subscriptionType is the same across requests.

At the same time, the values under customerAccounts have changed between requests (note customerAccounts.customerAccount[0].accountId being different). This means the payload validation should fail in the second request.

Current Behaviour

The function does not throw a validation error even though payload validation is enabled and parts of the payload differ from the stored one.

See screenshot below, where I send both requests and both are successful:
image

Code snippet

import os
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer,
    IdempotencyConfig,
    idempotent,
)
from aws_lambda_powertools.utilities.typing import LambdaContext

persistence_layer = DynamoDBPersistenceLayer(table_name=os.environ["TABLE_NAME"])
config = IdempotencyConfig(
    event_key_jmespath='["rcn", "subscriptionType"]',
    payload_validation_jmespath="customerAccounts",
)

@idempotent(persistence_store=persistence_layer, config=config)
def handler(event: dict, context: LambdaContext):
    return True

Possible Solution

As part of a feature added in #3446 we are using DynamoDB ReturnValuesOnConditionCheckFailure whenever available. This feature allows the DynamoDB to return an item that caused a conditional operation to fail, which in turn allows the Idempotency utility to get the IdempotencyRecord of a request without having to do an additional read.

In the base idempotency class (here), we have added an if/else statement that uses the returned item when available, or otherwise it calls the _get_idemptoency_record() method when one is not available - this last case can happen when customers are using older versions of boto3.

With the current implementation, the logic that validates the payload is executed only in the else clause, which causes payload validation to be skipped.

A potential fix would be to make the _validate_payload() method in the BasePersistenceLayer class and then call it on the exc.old_data_record, similar to this:

record = exc.old_data_record
if exc.old_data_record is not None:
  self.persistence_store.validate_payload(
    self.data,
    record
  )
  else:
    record = self._get_idempotency_record()

Steps to Reproduce

See steps above.

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.12

Packaging format used

PyPi

Debugging logs

No response

Metadata

Metadata

Assignees

Labels

cant-reproduceAny issues that cannot be reproducedidempotencyIdempotency utility

Type

No type

Projects

Status

Shipped

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions