Manipulation of Idempotent response

[royassis]

Use case

We have a client-server web application.
The server side is built using AWS SAM with an API Gateway and multiple lambda functions. The server side laso executes an asynchronous pipeline.

We want to prevent cases where a duplicated message is sent to the backend and triggers the pipeline multiple time.
We also want to notify the user if a duplication occurred, and hence his request was not processed, this is the main issue.

Solution/User Experience

Add an X-Idempotent: True header to the response.

Alternative solutions

A hook for changing the response (e.g. adding a field).

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[heitorlessa]

cc @walmsles as I'm sure you'll have ideas on DX here ;)

[leandrodamascena]

Original discussion: #2158

[walmsles]

Interesting - always interesting to hear more about different solution needs. The complexity around this is also that the idempotent handler runs as a Python decorator either on the lambda_handler or other function in the call chain and has zero knowledge of where it fits or where the event came from, which should be maintained.

We must also be clear that Idempotent responses are not just about API responses, so feel this is not a standard library feature that Powertools should provide "out of the box" in a canned way, as indicated by the initial request.

I think a mechanism for "intercepting" the idempotent response is useful and would provide @royassis with the capability to implement the customisation for his use case. This also provides future customisation opportunities for anyone. How the response is sent back is entirely up to the developer within the context of their solution, which is where control needs to be placed for any customisation of an idempotent response (in developer-maintained code).

Here is an example of what I am thinking using Powertools API Gateway resolver and an Idempotent Intercept at the API method level, which makes sense for an API use case (Note: I know an idempotent decorator for API methods is not really a thing today - but it should be and would be perfect for this use):

import json
from http import HTTPStatus

import requests

from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import (
    APIGatewayRestResolver,
    idempotent_api,
    Response,
    content_types,
)
from aws_lambda_powertools.utilities.idempotency import (
    DynamoDBPersistenceLayer, IdempotencyConfig
)
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.shared.cookies import Cookie
from aws_lambda_powertools.utilities.typing import LambdaContext

tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver()

persistence_layer = DynamoDBPersistenceLayer(table_name="IdempotencyTable")

def payment_intercept(payload : dict, idempotent_response: bool):
    response = Response(
        status_code=HTTP.OK.value,
        body=payload,
        content_type=content_types.APPLICATION_JSON,
        header={"X-Idempotent": idempotent_response}
    )

    return response

@app.post("/pay")
@idempotent_api(resolver=app, event_key_jmespath="powertools_json(body).[user, product_id]", persistence_store=persistence_layer, response_intercept=payment_intercept)
@tracer.capture_method
def process_payment():
    payment = create_subscription_payment(
        user=app.current_event.json_body['user'],
        product=app.current_event.json_body['product_id']
    )
    ...
    return Response(
        status_code=HTTP.OK.value,
        body=json.dumps(
            {
                "payment_id": payment.id,
                "message": "success",
            }
        ),
        content_type=content_types.APPLICATION_JSON
    )


@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)
@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

The above implies:

• idempotent_api() decorator is created and works (just thinking about it right now) - I see this as a useful thing and applying idempotency at the API resolution layer is easy to reason about if you have a monolithic API
• idempotency interceptor - would require an internal decorator to intercept idempotency response and a change to idempotency internally to return the payload AND whether the result was read from the Cache. This shouldn't break anything as the idempotency internal interceptor would hide this fact from existing use-cases.
• idempotency itself may need a newer class to unwrap API Response Objects for storing in the PersistenceProvider.

[heitorlessa]

got some security enhancements I need to prioritize first, but adding this to be looked after in the next iteration (next week).

Thank you everyone!!

[heitorlessa]

Haven't forgotten, our security enhancements are taking longer than I expected (to give you a glimpse of the complexity), plus COVID took a hit on me (back now 🙌).

Adding help wanted as we'd appreciate anyone taking a stab at this - @royassis feel free to give it a try and we can help refine it.

---

What we seem to be after:

• A mechanism to call an user-defined function for requests that are idempotent (think Cache: Hit -> hook)
• Optionally a new route parameter or decorator to more easily make a route idempotent and include a header as originally asked

I say, optionally, because it's only a matter of time until someone needs something else, and eventually it'll bloat in complexity and cognitive load to remember many parameters.

A hook-like would be useful to give full flexibility for customers to do as they please, and it'd only be called IF the transaction is idempotent - making it suitable for all sorts of use cases.

What we don't want

We should not modify the idempotent response of a decorated method. I think this is a better job for a deserializer (cc @ran-isenberg) in case you want to deserialize the JSON serialized idempotent response into something that better match your method return signature (e.g., Idempotency, Dataclass, Custom, etc.). Happy to hear counter arguments and learn from it too, specially how tricky it'd be from the typing system.

[ran-isenberg]

@heitorlessa I was thinking of adding a JSON serializer/deserializer optional arguments to the config class. That way I would have total control of the way it is saved to the database and loaded back from. What do you think?