RFC: Function Idempotency Helper

[igorlg]

Key information

• RFC PR: feat: Idempotency helper utility powertools-lambda-python#245
• Related issue(s), if known:
• Area: Utilities
• Meet tenets: Yes

Summary

Helper to facilitate writing Idempotent Lambda functions.
The developer would specify (via JMESPath) which value from the event will be used as a unique execution identifier, then this helper would search a persistence layer (e.g. DynamoDB) for that ID; if present, get the return value and skip the function execution, otherwise run the function normally and persist the return value + execution ID.

Motivation

Idempotency is a very useful design characteristic of any system. It enables the seamless separation of successful and failed executions, and is particularly useful in Lambdas used by AWS Step Functions. It is also a design principle on the AWS Well Architected Framework - Serverless Lens

Broader description of this idea can be found here

Proposal

Define a Python Decorator @idempotent which would receive as arguments a) the JMESPath of the event key to use as execution ID, b) {optional} storage backend configuration, e.g. DynamoDB table name, or ElasticSearch URL + Index).

This decorator would wrap the function execution in the following way (pseudo-python):

from aws_lambda_powertools.itempotent import PersistenceLayer

def idempotent(func, event_key, persistence_config):
  def wrapper(*args, **kwargs):
    persistence = PersistenceLayer(persistence_config)
    key = jmespath.find(event_key, **kwargs['event'])
    persistence.find(key)

    if persistence.executed_successfully():
      return persistence.result()

    try:
      result = func(*args, **kwargs)
      persistence.save_success(key, result)
      return result
    except Exception => e:
      persistence.save_error(key, e)

  return wrapper

Usage then would be similar to:

from aws_lambda_powertools.itempotent import itempotent

@idempotent(event_key='Event.UniqueId', persistence_config='dynamodb://lambda-idemp-table')
def handler(event, context):
  # Normal function code here
  return {'result': 'OK', 'message': 'working'}

The decorator would first extract the unique execution ID from the Lambda event using the JMESPath provided, then check the persistence layer for a previous successfull execution of the function and - if found - get the previous returned value, de-serialize it (using base64 or something else) and return it instead; otherwise, execute the function handler normally, catch the returned object, serialize + persist it and finally return.

The Persistence layer could be implemented initially with DynamoDB, and either require the DDB table to exist before running the function, or create it during the first execution. It should be in such way as to allow different backends in the future (e.g. Redis for VPC-enabled lambdas).

Drawbacks

This solution could have noticeable performance impacts on the execution of Lambda functions. Every execution would require at at least 1, at most 2 accesses to the persistence layer.

No additional dependencies are required - DynamoDB access is provided by boto3, object serialisation can use Python's native base64encode/decode

Rationale and alternatives

• What other designs have been considered? Why not them?
  No other designs considered at the moment. Open to suggestions.

• What is the impact of not doing this?
  Implemention of idempotent Lambda functions will have to be done 'manually' in every function.

Unresolved questions

• How to make the persistence layer access as fast as possible?
• Which other persistence layers to consider (DynamoDB, ElasticSearch, Redis, MySQL)?

[michaelbrewer]

Some comments:

• We should handle duplicate calls, and short circuit with a in progress error. (This would mean one additional write to create the initial lock and a second right when the task has been completed). For Dynamodb could use conditional conditional expression to “insert or get” the saved request
• We should allow for composite keys. Like an request id and the authenticated user id (like a cognito sub). This would prevent users from accessing other event results.
• Could we leverage Lambda extensions to offload the saving of the result to dynamodb?
• There should be an optional TTL on these records, for example we cache the result for up to 24 hours.
• In same cases what we store in dynamodb might be sensitive. So an optional flag for field level encryption would help
• We might also want to validate if the same idempotent key has been used for different request parameters. This would prevent misuse of this feature by the end user. Either the end user is not generating a unique key or are reusing the same key for a different request. To support this we would also optional store the request parameters.

[michaelbrewer]

Sorry one more thing to consider.

When there is an error, sometimes we might want to allow for retrying by the same idempotent key. For example, the lambda is calling some downstream service that is temporarily unavailable, calling with the same idempotent could make another downstream request. In this case we would want to remove any lock record that might have been created.

However if the end user is passes in an invalid request we should return a 400 type of response and potentially not do a round trip to the persistence layer. This will prevent bad requests from taking up any additional bandwidth.

Otherwise, we could fall back to an in memory local fifo cache to serve back just the most recent retries.

I feel like if we can some how offload the dynamodb writes to an async task, then we can optimize for the common case of the first request going through.

And we should also consider using metric or annotations to track cache hits, so that the developer can see how useful this feature really is.

This feature is particularly useful for financial transaction that can take some time to process and you don’t want to double charge the end user.

[michaelbrewer]

Couple features we could pair with this is an idempotent queue. There we want to do some validation and checks, before putting on one request into the dynamodb or some other queue. Idempotent keys would prevent double submits, and the heavier processing can be handled by another process or lambda. Having the lock record at the beginning of the request can work as well for when our lambda execution can run much longer than the request timeout of the caller.

As a separate RFC we could have a nonce feature, which is similar to this, but instead we return a single use token. (But that is for another occasion 😄 )

[igorlg]

Great comments, @michaelbrewer ! Some inline replies.

• We should handle duplicate calls, and short circuit with a in progress error. (This would mean one additional write to create the initial lock and a second right when the task has been completed). For Dynamodb could use conditional conditional expression to “insert or get” the saved request

Yes! I'll amend the flow description, it should be something like "Start -> Extract Key -> Check in DB if running or completed -> If running, return 'BUSY'; if complete, return saved result; else Insert 'RUNNING' -> Continue"

• We should allow for composite keys. Like an request id and the authenticated user id (like a cognito sub). This would prevent users from accessing other event results.

In general terms, the developer needs to provide a uniquely identifiable value for each execution. If that requires a composite key, by all means. I only worry about the complexity of this implementation for v0.1

• Could we leverage Lambda extensions to offload the saving of the result to dynamodb?

• There should be an optional TTL on these records, for example we cache the result for up to 24 hours.

Agree 100%. This factors in the table design, particularly if we allow the solution to 'create table if not exists'

• In same cases what we store in dynamodb might be sensitive. So an optional flag for field level encryption would help

Agree 100% as well. Optional config flag to KMS encrypt fields. Worth mentioning the (possible) additional cost and time in the docs.

• We might also want to validate if the same idempotent key has been used for different request parameters. This would prevent misuse of this feature by the end user. Either the end user is not generating a unique key or are reusing the same key for a different request. To support this we would also optional store the request parameters.

Not sure I follow. The only requirement for the idempotence key is that it uniquely identifies each execution. If requests are made with different parameters, they should have different idemp. keys...

• When there is an error, sometimes we might want to allow for retrying by the same idempotent key. For example, the lambda is calling some downstream service that is temporarily unavailable, calling with the same idempotent could make another downstream request. In this case we would want to remove any lock record that might have been created.

Yes. Few options here. If the function throws an Exception, we can:

• a) Record that information in the DB (e.g. key=xyz, status=FAILED) and then, on the next execution w/ key=xyz, we know to continue running, or
• b) Delete the record w/ key=xyz, so next time it'll be like a first execution, or
• c) Expose a few exception types that dictate the behaviour for next executions (retry or not).

• However if the end user is passes in an invalid request we should return a 400 type of response and potentially not do a round trip to the persistence layer. This will prevent bad requests from taking up any additional bandwidth.

Absolutely, best to catch invalid requests as early as possible.

[michaelbrewer]

@igorlg - i agree that some of this feedback is based on more advanced use cases.

Use case:
Lambda is part of an api gateway function used by external developers (or internal developers but on a different team). So we would want to protect against usage errors. Like reusing idempotent keys for different requests. For V0.01 we could just document that the request id has to be server generated using UUID. And retries should be for the exactly same request.

Use case:
Lambda can be called via a Appsync resolver or Api gateway directly. So to help prevent a bad user reading a cached response for another user, a compound key taking the user inputted fields along with authenticated fields. (Again we could defer this to the documentation).

Another interesting feature when pairing this with the API gateway would be to return a header parameter like Idempotent-Replayed: true to signify whether the response was a cache hit or a replay.

Stripe has a pretty good implementation of idempotency - https://stripe.com/docs/idempotency

[heitorlessa]

all great comments - my suggestion is to expand the RFC @igorlg to clarify on:

• Responding to duplicate requests (e.g. callback function like batch utility, return last response cached, depending on event source respond X)
• Responding to exceptions (e.g. clear cache, retry/do-not-retry)
• Standalone function for more control over failure mode
• Persistence layer should follow the Parameters implementation of Bring Your Own - Implement one, DynamoDB, and leave others for post-launch
• Separate what's must have vs nice to have
• Sequence diagram or text (step->step) to quickly visualize steps
• Don't worry about naming of the parameters or pseudo code - We can handle UX later after we get down to the must haves

As to comments by @michaelbrewer on sensitive fields, composite keys, Extensions and metrics, here's my initial take on:

• Sensitive fields and Extensions - Customers can customize their persistent layer on inserting and querying the data. In the case of Extensions, they can query filesystem, use IPC, etc.
• Composite keys - JMESPath allows for multiselect. If the return from the query selection is a Sequence or a Dict, we can serialize it and encode/hash it
• Metrics - If we support a callback fn (e.g. idempotency_handler()), customers can use Metrics utility or their own mechanism to track that. We'd call this function when the request is idempotent and when using the decorator, so they'd have full control over the logic, how the response should be returned given their event source (e.g. API GW vs SFN)

[michaelbrewer]

@heitorlessa should we allow for an in-memory LRU cache?

Also for the in-progress errors, we might want a overall timeout (or permanent failure), but we can't always assume the original lambda was able to run to completion, either to memory or lambda timeout.

I think there will be more docs than code for this? And a set of recommendations and warnings about cost.

Stripe wrote a couple good articles, and i am sure their are AWS ones out there too.

[heitorlessa]

@michaelbrewer quite possibly along with a TTL and max LRU cache size - A dict fits the purpose here.

Not entirely sure yet on upon-exception behaviour whether we want to burst the LRU cache and call a provided function e.g. idempotency_handler. That is why my ask to Igor to expand on responding to duplicate events as well as exceptions.

If you have some of those articles or ideas please bring it on ;)

[mwarkentin]

One of the stripe devs has some good articles (somewhat PG related though):

https://brandur.org/idempotency-keys

https://stripe.com/blog/idempotency

[michaelbrewer]

@mwarkentin - yep, those are the articles i was referring to.

I know initially we don’t want to bloat v0.01 of this feature, but i would either flag it as a alpha api with possible breaking changes, or factor in where this could go to in the api design.

For example implementing this feature various based on how it is being integrated. In the context of the API gateway or AppSync we might want to have help functions to return errors, http headers and http error codes etc.

[michaelbrewer]

@heitorlessa maybe we should create a shared repo for RFCs and docs shared by the Typescript/Python and Java implementations, so we keep the design in sync and help other maintainers to support the implementation up to date.

[michaelbrewer]

@heitorlessa @nmoutschen should we collate these requirements and start on a prototype implementation?

[heitorlessa]

Sorry folks I'm unwell but this is being taken care of -- cc @cakepietoast to provide an answer

I'll reply to the shared RFC repo as soon as I'm recovered

[to-mc]

Get better soon @heitorlessa! I'm working with @igorlg on this, we expect to have at least the starting point of an implementation added to the repo in the next week or so based on this RFC.