RFC: idempotency utility

[walmsles]

Description of the feature request

Problem statement
Idempotency is a core Cloud issue that needs a solution to enable stable, fault-tolerant systems that can be affected by repeated transactions. A true idempotency solution as a utility for Powertools would be really useful for this project based on the function/features of the existing AWS Lambda Powertools for python

Summary of the feature
Link to Python Lambda Powertools documentation as a good example to follow: https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency

Benefits for you and the wider AWS community
Supply a good idempotent solution "Out of the Box" with Powertools for Typescript developers.

Describe alternatives you've considered
Looked at this: https://www.npmjs.com/package/middy-idempotent but relies on redis for storage and the Idempotent Key generation is not ideal and looks like it will be problematic.

Additional context
https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/

Related issues, RFCs

None.

[dreamorosi]

Hi @walmsles thank you for the request. Feature parity with the Python version is definitely something that we have on our radars. Middy-compliant middlewares are only one of the usages that we want to cover at this stage (with the others being Class method decorators & manual instrumentation) so the feature deserves a closer look.
Please give us some time to discuss this internally.

[ijemmy]

Hi everyone, here is a design proposal. Would appreciate your comments on this (especially around the Utility interface as it's different from Python and Java which have no constraint on decorator usage.)

Design proposal (Request for comments)

1. Summary

The goal of this document is to propose the scope and design of Idempotency utility for Powertools for TypeScript. The utility has been implemented in the Python and Java version. We will use the current Python implementation (https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/) as a baseline, and describe only the differences we will make in TypeScript. Anything not discussed here will be the same as in Python version.

This RFC assumes that you are familiar with Python’s implementation. If you aren’t, please check the documentation of Python Idempotency utility (https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/) first.

2. Motivation

Idempotency is a common pattern used by many customers. It guarantees that any retry with the same “idempotency key” should not be executed again. This utility aims to provide “out-of-the-box” idempotency on the top of Lambda with minimum code from library user.

With this utility, customers only need to provide add a decorator, Middy middleware, or a function wrapper on the top of business logic functions. Customers may customize the utility behavior by providing different config, or provide a custom PersistentLayer to use different persistent storage for storing idempotency key.

3. Utility interface

There are two usages of using Idempotency utility.

1. Idempotency at the handler level (via Middy middleware or decorator)
2. Idempotency in a function called inside the handler function. (via decorator or function wrapper)

The second option is useful for batch or multi-record processing. Imagine that we receive 10 records in a single request. We want the idempotency at the record level, not at the handler level. We can loop through the records and call the decorated function. We need to specify which

Unlike existing utilities, the "manual" options are complex and expose a lot of implementation. Thus, the first release will have less usage options than those utilities.

3.1.1. Idempotent handler (via Middy middleware)

import { 
  makeHandlerIdempotent,
  DynamoDBPersistenceLayer,
  IdempotencyConfig
} from '@aws-lambda-powertools/idempotency';
import middy from '@middy/core';

const config = new IdempotencyConfig({...});
const ddbPersistenceLayer = new DynamoDBPersistenceLayer({...});

const lambdaHandler = async (_event: any, _context: any): Promise<void> => {
  /* ...Function logic here... */
}

export const handler = middy(lambdaHandler)
  .use(makeHandlerIdempotent({
    config: idempotencyConfig,
    persistenceLayer: ddbPersistenceLayer,
  });

3.1.2. Idempotent handler (via Decorator)

import { 
  idempotentHandler, 
  DynamoDBPersistenceLayer,
  IdempotencyConfig
} from '@aws-lambda-powertools/idempotency';

const config = new IdempotencyConfig({...});
const ddbPersistenceLayer = new DynamoDBPersistenceLayer({...});

class Lambda implements LambdaInterface {
  // Decorate your handler class method
  @idempotentHandler(config, ddbPersistenceLayer)
  public async handler(_event: any, _context: any): Promise<void> {
    /* ...Function logic here... */
  }
}

export const handlerClass = new Lambda();
export const handler = handlerClass.handler.bind(handlerClass);

3.2.1 Idempotent function (via Decorator)

Note that this feature is typically used with Batch Utility (https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/#idempotent_function-decorator). Given that we don’t have Batch Utility yet, we will give an example with a simple for loop. But be mindful that the more resilient way is to handle failed records, which is out of scope for this utility.

import { 
  idempotentFunction, 
  DynamoDBPersistenceLayer,
  IdempotencyConfig
} from '@aws-lambda-powertools/idempotency';
import middy from '@middy/core'

const config = new IdempotencyConfig({...});
const ddbPersistenceLayer = new DynamoDBPersistenceLayer({...});

class Lambda implements LambdaInterface {

  public async handler(_event: any, _context: any): Promise<void> {
    
    const records = /*..Extract SQS/DDB stream record, etc..*/
    
    const results = []
    for(record of records) {
      results.push(this.process(record));
    }
    
    /* ...Format and return result... */
  }
  
  @idempotentFunction({
    dataKeywordArgument = 'record', // Match with param name of decorated function below
    config, 
    ddbPersistenceLayer
  })
  private process(record: any) {
    /* ...Function logic here... */
    return result;
  }
    
}

export const handlerClass = new Lambda();
export const handler = handlerClass.handler.bind(handlerClass);

3.2.2. Idempotent function (via wrapper)

Majority of TypeScript/JS Lambda code are not using a class. Middy middleware is not an option as we are dealing with a non-handler function. To support this major use case, we provide a wrapper function

import { 
  makeFunctionIdempotent, 
  DynamoDBPersistenceLayer,
  IdempotencyConfig
} from '@aws-lambda-powertools/idempotency';

const config = new IdempotencyConfig({...});
const ddbPersistenceLayer = new DynamoDBPersistenceLayer({...});

 /**
 * Function to process a single record
 */
function processRecord(record: any) {
  /* ...Function logic here... */
  return result;
} 

/**
 * Higher-order function to process a single record
 */
const processIdempotently = makeFunctionIdempotent(
  processRecord, 
 {
    dataKeywordArgument: 'record', 
    /*... other options...*/
 }  
);

const lambdaHandler = async (_event: any, _context: any): Promise<void> => {
    
  const records = /*..Extract SQS/DDB stream record, etc..*/

  const results = [];
  for (const record of records) {
    results.push(processIdempotently(record));
  }
  /* ...Format and return result... */
}

//...

4. Requrements

4.1 Feature breakdown

The first release will contain only high/medium priority features. All "Out of scope" features won’t be implement in the future releases unless there is a clear signal from customers.

Feature	Priority	Description	Note
FR1	High	Provide idempotency at handler level (via Middy middleware and decorator), including edge cases
FR2	High	Provide idempotency at function level (via wrapper function and decorator), including edge cases
FR3	High	Return the same result when called with the same payload
FR4	High	Can customize time window	expires_after_seconds in Python
FR5	High	Can parse data via JMESPath. We will use external library for JMES parsing for MVP	event_key_jmespath=powertools_json(body). in Python he alternative solution is to have config to pass a function to extract data. But this will be inconsistent with other Powertools. We can reuse jmespath.js
FR6	Low	Payload validation for the case that two requests with the same idempotency key may contain different payload. See doc for details.	payload_validation_jmespath in Python
FR7	Low	Local (inside Lambda) caching with LRU cache	use_local_cache and local_cache_max_items in Python
FR8	Medium	Passing configuration to AWS SDK V3
FR9	Medium	Support a composite primary key (for reusing the same table by multiple functions)	sort_key_attr and static_pk_value in Python (for DDB persistence layer)
FR10	Medium	Throw an exception if the idempotency key isn't found	raise_on_no_idempotency_key in Python
FR11	Out of scope	Filter sensitive field in the result from being cache	Customers can do this by customizing the persistent layer
NFR1	High	Allow customization of persistence layer through extending a base class or an interface
NFR2	Low	Switching hash function	hash_function in Python
NFR3	High	Handle error thrown by the wrapped function. The idempotence record in persistent should be cleared.	See more discussion on this on the "Edge cases" section below.
NFR4	High	Handle Lambda timeout	https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/#lambda-timeouts
NFR5	Low	Idempotency metrics (e.g. error by type, cache hit, etc.)
NFR6	Out of scope	Support recovery point for Lambda function perform more than one step and fails midway . The next request continues from the recovery point instead of starting from the beginning (and cause the same side effect twice)	We will assume that the handler has a single side effect. There is no left state that needs to rollback when it fails.

Info
Note: feature with high/medium priority won't be included in the first release (MVP). |  

4.2 Edge cases

Idempotent function contains many edge cases and limitation. We will use the same behavior as Python and Java version. This section clarifies the behaviours of non-happy flows for reference during implementation.

1. What if the second request with the same key when the first request is in progress?
   Similar to Python version, we will throw IdempotencyAlreadyInProgressException (https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/#handling-concurrent-executions-with-the-same-payload). The client side will receive 5xx from uncaught exception in a Lambda handler. . We may provide option to customise this behaviour in the future but it’s out of scope for now.

2. What if two same requests come in (almost) at same time? Will both of them pass the initial check and get executed twice?
   When a request comes in, we will try to save an in-progress records first. If the save succeeds, we continue, if it fails, we handle the error. In this situation, the saving of one request will fail. How this happens depends on the implement of PersistentLayer. For the default (DDB) one, we use a conditional attribute_not_exists put_item request.

3. What if the second request with the same key arrives when the existing record has expired?
   The existing result & status of the previous idempotency record will be overwritten with the new in-progress record.

4. What if the code throw an uncaught exception?
   The utility will remove the existing in-progress record from persistent layer, allowing client to call Lambda again with the same key.
   If customers prefer otherwise, they can catch the exception themselves and return appropriate result. In that case, the next request (within TTL), will be use the cached result. This may be the case for invalid requests (e.g. 4xx error).

5. What if the function has multiple side effects? The first request may be partially-executed, and has already caused a side effect. Howerever, it fails to complete and throw an exception?
   We won’t support multiple side effects. This is the same with Python version (check the first note in Python Idempotency Request Flows (https://awslabs.github.io/aws-lambda-powertools-python/latest/utilities/idempotency/#idempotency-request-flow) )

   If customers choose to do this (not recommended), the responsibility belongs to the Lambda handler. It needs to properly catch the exception, and clean the side effects previously done. Then it can rethrow an exception to idempotent utility to delete the idempotency record.

   An alternative option is to make our utility support recovery point (NFR6). Let’s say that the handler function has two steps and both of them have side effects. We can put a recovery point between two steps. When the first step completes, we update the idempotency record. If the 2nd step fails and get retried, the function can skip the first step.
   However, this complicates the design and interface of our utility. In addition, the system also may be left with a consistent state if retry requests never reaches the Lambda function. The better design is probably split two side effect to two Lambda functions and coordinate them with a Step Function or have SQS queue in between.

6. What if the Lambda function timeout?
   When calculating TTL of the record, we will look at Lambda’s context object to determine timeout. If the timeout is shorter than configured TTL, we will use this value for TTL field instead. See implementation in PR #1387 (https://github.com/awslabs/aws-lambda-powertools-python/pull/1387/files)

7. Can idempotency utility itself throws an exception? What are them and how should customers handle it?
   Below’s a list of possible exception thrown by the utility.
   Customers cannot (and should not) handle these exceptions. They will be bubbled up to Lambda service.

   1. IdempotencyItemAlreadyExists Thrown when there is already an in-progress and unexpired idempotency record. This can happen if the 2nd request comes before the first one has completed and updated the record with a result.
   2. IdempotencyItemNotFound Thrown when it cannot get the record from persistence layer. This shouldn’t happen in normal circumstance.
   3. IdempotencyInvalidStatus Thrown when the record’s status value is wrong. This indicates a bug in persistence layer implementation
   4. IdempotencyInconsistentState Thrown when the record status is incorrect. (e.g. the in-progress record is somehow expired after its initial save). The utility catches this exception at the top level and retry once before rethrowing the exception
   5. IdempotencyValidationError Thrown when the payload validation (FR6) is enabled, and validation fails.
   6. IdemptoencyKeyNotFound Thrown when an idempotency key not fund in the payload, and it’s configured to throw an exception in thsi case (FR10)

5. Opened discussions and decisions

This is a list of important decision points from maintainer discussion. Subject to changes from comments from community.

1. 1. Should we implement JMESPath helper utility or reuse an existing library?
      For the MVP (initial release), we will use an existing JMESPath library. I haven’t done a research on this but the criteria will be popularity (commonly used), and package size. This is to make the release fast and focus on the main issue we’re solving (idempotency).
2. Should we have local (result) caching feature in the first release?
   We will go without this feature in MVP unless there is a clear customer need during RFC review.
   The benefit (performance gain by saving a single call to DDB) may outweights the complexity at the first stage because (1) It only applies to warm Lambda (2) the request may be routed to another execution context, the cache adds no benefit with additional latency.The most efficient use of this feature is through provisioned concurrency.
3. Should we support composite key in the first release?
   Yes, it’s required for sharing the same DDB table with multiple idempotent Lambda functions (imagine 1k of functions).
4. What if the result contains sensitive data?
   Customers can customize the persistent layer class for this. If there is a lot of requests/questions, we can provide an option to simply save/not save result. It’s going to be complex to mask only some fields. (We need to provide a maskingFunction or JMESPath).
5. JavaScript SDK V2 or V3?
   We’ll go for V3 (for all utilities). V2 has the only advantage of already in the run time. But that is not the recommended approach. V3 supports tree shaking (only import the required code path), so we think that the performance gain for cold start isn’t large. This also eliminates migration work in the future.

6. Implementation deviation from Python

80-90% of the implementation will be based on the Python version. However, we will deviate where it’s appropriate as we don’t have a constraint of making breaking changes yet.

1. Any Pythonic convention will be changed to TypeScript’s (e.g. all names with raise will be thrown)
2. The initial release will include the fixes for current unmerged breaking-change issues:
   1. Computing hash key with module + qualified name #1330 (Computation of idempotency hash key using unqualified name  powertools-lambda-python#1330)
   2. Fix timeout issue for idempotent_function
3. We may reconsider the responsibility of each class or introducing new ones. For example:
   1. In Python, @idempotent_function has no access to the context object. Thus, customers have to call register_lambda_context on the config so that it can handle timeout properly. This can be error-prone as some may forget and recieve warning (line 360 (https://github.com/awslabs/aws-lambda-powertools-python/pull/1387/files#diff-becc2a8b1233bddbb0d38b13a8634ab96bdd5d00f16f0514f6ea1df509c3b1d6R360)). We can force this in the new design by subclassing a separated config class for @idempotent_function, and makes context object a required config. Customers will find an error at run time.
   2. In Python, the BasePersistenceLayer class also performs JMES validation and key extraction. This can be done in upper layer first.

7. References

1. Orignal RFC from from Python version
   1. RFC: Function Idempotency Helper · Issue #28 · awslabs/aws-lambda-powertools · GitHub (RFC: Function Idempotency Helper powertools-lambda#28)
2. Original implementation PR
   1. feat: Idempotency helper utility powertools-lambda-python#245
3. Timeout discussion on Python version
   1. Issue: Idempotency and Lambda timeouts · Issue #1038 · awslabs/aws-lambda-powertools-python · GitHub (Idempotency and Lambda timeouts powertools-lambda-python#1038)
   2. PRs: feat(idempotency): handle lambda timeout scenarios for INPROGRESS records by rubenfonseca · Pull Request #1387 · awslabs/aws-lambda-powertools-python · GitHub (feat(idempotency): handle lambda timeout scenarios for INPROGRESS records powertools-lambda-python#1387)

[AlessandroVol23]

That sounds very nice and is indeed much needed 👍

We're also currently simply using a middy middleware to check if a request was already executed and add a flag to the context object but things like

• Error handling
• retries
• Early exit
• Handler & Function level

Would definitely help a lot. Thanks!

[jeffrey-baker-vg]

This looks great! We are very excited that this utility has been prioritized!

After reading the design proposal, we do have a question and a couple of comments.

First, the question: We noticed that FR3 is "Return the same result when called with the same payload", which made us wonder about the scope of FR1. Is it just "don't re-process given the same payload"?

The comments:

1. We suggest switching the order of FR1 and FR2. The function wrapper seems like the most extensible part of this library, and there is a lot of value in that alone. This would immediately provide value for event-driven systems, and we think it would be the easiest thing to implement first.
2. Related to the comment in 6.3 about deviation from Python, we recommend that you not make the lambda context a required configuration. While we understand that this is a library ultimately intended for lambdas, the function-level idempotency feature is very extensible beyond that as long as lambda context remains optional.
3. If lambda context remains optional in the function-level idempotency feature, then this requirement can potentially be broken into even smaller MVPs by limiting the number of edge cases that need to be considered.