Feature request: Be able to have more flexibility with how the idempotency key is being handled

[wizardone]

Use case

Hello folks, while migrating a lambda function to use powertools I really needed to have slightly more control over the idempotency key and after reading the docs and code extensively I don't think there is currently support for what I'm after. In a nutshell:

• I would like to be able to configure whether or not I apply a hash function
  • That would assume I have preconfigured the required idempotency key to what I need and the library can just use it.
• Be able to specify the digest for the generated hash
  • Allows me to specify hex if I wanted to.

Both options are interconnected for my particular use case. In other words if I'm able to specify the digest, then powertools will give me everything I need. Otherwise I would like to have the freedom to generate what I need myself and use it.

I appreciate if that level of control is not something that you'd like to provide, my use cases are very specific and powertools already gives flexibility. Still, let me know if either option makes sense to you. Thank you.

Solution/User Experience

Implementing giving the option to set digest can be as low touch as adding a new configuration option to the IdempotencyConfig:
pseudo code:

baseDigest: 'hex' // default to base64

private generateHash(data: string): string {
    const hash: Hash = createHash(this.hashFunction);
    hash.update(data);
    const digest = this.baseDigest ?? 'base64'
    return hash.digest(digest);
}

Implementing the option to hash or not might be slightly more involving and I'm not aware of the nuances of the code, but a very rough implementation could be:

private getHashedIdempotencyKey(data: JSONValue): string {
    const payload = this.eventKeyJmesPath
      ? (search(
          this.eventKeyJmesPath,
          data,
          this.#jmesPathOptions
        ) as JSONValue)
      : data;

    if (BasePersistenceLayer.isMissingIdempotencyKey(payload)) {
      if (this.throwOnNoIdempotencyKey) {
        throw new IdempotencyKeyError(
          'No data found to create a hashed idempotency_key'
        );
      }
      console.warn(
        `No value found for idempotency_key. jmespath: ${this.eventKeyJmesPath}`
      );
    }
    const idempotencyHash = this.suppliedHashedKey ? this.generateHash(JSON.stringify(deepSort(payload))) : this.suppliedHashedKey
    return `${this.idempotencyKeyPrefix}#${idempotencyHash}`;
 }

Alternative solutions

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @wizardone, thank you for opening this feature request and for taking the time to propose alternatives.

Before moving forward with the discussion, I'd like to understand your use case a bit better.

I am not asking you to disclose anything specific to your workload, but rather understand the root cause of wanting to have full control over the idempotency key.

My question comes from two angles: 1/ understanding whether what you want to achieve can be done in other ways with the current API, or 2/ if not, understanding whether the use case/need is broad and generic enough that it might be useful for other customers - this last, in turn, will play a significant part in our decision of implementing this or not.

Thank you in advance, also in the meanwhile, would you mind taking a look at this other feature request (#3515) that is already being implemented, to see if it might cover part of all your needs?

[leandrodamascena]

Hey @wizardone thanks a lot for opening this and bringing up some ideas for your specific use case! In addition of what @dreamorosi already told, I have some special concerns here about the Idempotency mechanism.

• I would like to be able to configure whether or not I apply a hash function

  • That would assume I have preconfigured the required idempotency key to what I need and the library can just use it.

If I'm understanding correctly, and please correct if I'm wrong, you want to have an experience like this:

const createSubscriptionPayment = makeIdempotent(
  async (
    transactionId: string,
    event: Request
  ): Promise<SubscriptionResult> => {
    // ... create payment
    return {
      id: transactionId,
      productId: event.productId,
    };
  },
  {
    persistenceStore,
    dataIndexArgument: 1,
    config,
    myIdempotencyKeyWithHash: "myKeyWithHash"
  }
);

In this case, if you use the myIdempotencyKeyWithHash parameter, we'll use it directly as the IdempotencyKey in DynamoDB, ignoring the payload/fields/part of payload to calculate this key. If this is correct, I think this approach is more like caching the function result rather than doing idempotency, as it doesn't consider the payload's content anymore.

The AWS Well-Architected Framework defines idempotency in the chapter REL04-BP04 Make mutating operations idempotent as follows:

An idempotent service promises that each request is processed exactly once, such that making multiple identical requests has the same effect as making a single request.

So, in this case you can create an artificial key and no matter the payload you will always have the same key, since we will no longer calculate the IdempotencyKey based on the payload, making this function response cached and no matter the payload.

I'm not sure we should go down that route and make this available to customers. We need to have some guardrails in place.

• Be able to specify the digest for the generated hash

  • Allows me to specify hex if I wanted to.

Can you explain a bit more why you need to control the digest mechanism? We create a digest, which is a small value/signature of the entire message, and in our case it is a one-way process, making it irreversible. After creating the digest, we just use it to query DynamoDB in subsequent invocations, we don't roll it back to original value.

Again, please let me know if I am wrong in my understanding and I'm more than happy to discuss this solution.

[wizardone]

Thank you both for your responses.
Unfortunately I cannot share too much details about what I'm building, but I can give you a very similar example, which illustrates the principle:
I have 2 separate consumers, reading data from the same source. Newer one is using powertools and the old one does not. I need to ensure that idempotency is "shared" between them, aka it behaves exactly the same, thus if both consumers pick up the same data, only one will succeed. Unfortunately the old one uses a custom built idempotency solution and I need to bend the new one to behave in the same way.

Powertools is great and it mostly gets me covered: I can specify the dynamo table, primary key, etc, but in the end if the digest is different I'm lost :)
Now to answer the question you are about to ask: can I change the existing custom solution to play ball, instead of trying to change powertools? There's a wide range of factors which may turn this into a bittersweet experience for me, but if I have no other option I can go down that road.
As I said in the beginning I'm dealing with edge case scenarios, which are probably not widely covered. I appreciate your time and am happy to take suggestions if what I'm after is achievable with the current release of powertools. Thanks :)

[leandrodamascena]

Hello @wizardone! Thanks for telling us about your use case. What you shared is more than enough and for me this is a very interesting use case for Idempotency. I must be honest, I never thought about integrating two Idempotency implementations and make it work, nice!

While we can’t add this feature to our codebase for some of the concerns we’ve raised before, we're working on a new feature that gives you more control over your idempotency keys prefix. Instead of always using function_name#hash, you'll be able to set any prefix you want. When it get merged, you can extends the DynamoDBPersistenceLayer class and modify the getHashedIdempotencyKey method.

Keep in mind that getHashedIdempotencyKey is a protected method and not a public one. We don't expect to change this implementation any time soon, but we can't guarantee stability for this method since it's not public.

import { DynamoDBPersistenceLayer, idempotent } from '@aws-lambda-powertools/idempotency';
import { Context } from 'aws-lambda';

class MyOwnPersistenceLayer extends DynamoDBPersistenceLayer {
  protected getHashedIdempotencyKey(data: JSONValue): string | null {
    return this.idempotencyKeyPrefix; // When working with keyPrefix, this will return always what you set as keyPrefix and will ignore the payload calculation.
  }

const persistenceLayer = new MyOwnPersistenceLayer({ tableName: "ddbidempotency" });

const lambdaHandler = idempotent(
  async (event: Record<string, any>, context: Context) => {
    return "ABBB";
  },
  {
    persistenceStore: persistenceLayer,
    keyPrefix: "aaaa"
  }
);

export { lambdaHandler };

My TypeScript skills isn't the best and you'll probably need to make some small changes to this example, but consider this as a baseline to get some insights on solving your challenge.

Please let us know if we can help with anything else.