Feature request: DynamoDb idempotency causes hot partions

[elliottohara]

Use case

We have an existing dynamodb table that we're using for our system that has a composite key (pk, sk)... When you set the sortKeyAttribute, we end up with all idempotency checks for a given lambda sharing a partition - resulting in hot partitions.

Solution/User Experience

Ideally we'd like the ability to inject or override the getKey method so we can control how the row is created, or just change the basic implementation to not create hot partitions with busy lambdas.

Alternative solutions

Acknowledgment

• This feature request meets Powertools for AWS Lambda (TypeScript) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and .NET

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #typescript channel on our Powertools for AWS Lambda Discord: Invite link

[dreamorosi]

Hi @elliottohara thank you for opening this issue and proposing this idea.

I understand the problem we're trying to avoid, but I am unsure how it would look like or what's the best way to address it yet.

Assuming you were able to provide your own logic, how would you like to implement it/customize it?

Having this example would help us understand if it's something we can improve internally and provide it as an integrated experience, or if it's something specific to your implementation - in which case we'd just make the method easier to extend/override.

Thank you!

[elliottohara]

Hi @elliottohara thank you for opening this issue and proposing this idea.

I understand the problem we're trying to avoid, but I am unsure how it would look like or what's the best way to address it yet.

Assuming you were able to provide your own logic, how would you like to implement it/customize it?

Having this example would help us understand if it's something we can improve internally and provide it as an integrated experience, or if it's something specific to your implementation - in which case we'd just make the method easier to extend/override.

Thank you!

If I where doing it, I'd just set the partition key to the same thing that the Id is on a table without a composite key and set the sort key to (anything really, but probably what the sort key currently is).

I struggle to see the value of being able to query all idemptotency records for a given lambda (which is what the current solution provides) what I'm proposing simply means you can't get that data without a scan (which is fine by me).

[dreamorosi]

I see, thanks for clarifying.

Does setting a prefix at the function level using this parameter help reduce the pressure on the partition? At the very least with this you can have one partition for each function.

Alternatively, just to understand better, is using a table without sort key an option?

[dreamorosi]

On second thought - what I suggested with the prefix would work only if you're sharing the same table with multiple Lambda functions.

If you only have one function, then it won't change anything.

[dreamorosi]

I struggle to see the value of being able to query all idemptotency records for a given lambda (which is what the current solution provides) what I'm proposing simply means you can't get that data without a scan (which is fine by me).

This key design is meant to allow querying a specific item with a single GetItem operation - the fact that you can also query all the records for a function is just a side effect.

Just to clarify, here we're specifically talking about composite keys. If you can use a single key in your table, none of this is a problem and the Idempotency utility will generate a unique PK based on your payload hash.

If instead you're using a composite key, which is optional, then this is the best key design we could come up with while having the requirement of being able to query an item in a single operation. Keep in mind that once your table has a composite key DDB only allows you to do a GetItem/PutItem using both.

Now with this in mind, since we derive the sk from your payload and Lambda is stateless (aka there's no guarantee that your next request will be served by the same environment) then the pk must be static, or we wouldn't know how to query that item back.

If we were to introduce scan operations just to check if an item exists, we would need to scan the table for every request, which is bad for both costs and performance.

With that said, if you have any suggestions on how to improve this we'd be happy to do so.