Skip to content

Commit 6152b6d

Browse files
committed
docs(idempotency): cleanup dynamodb required resource; break subsections and update nav
1 parent f796f92 commit 6152b6d

File tree

1 file changed

+24
-18
lines changed

1 file changed

+24
-18
lines changed

docs/utilities/idempotency.md

Lines changed: 24 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -69,21 +69,25 @@ We provide Infrastrucure as Code examples with [AWS Serverless Application Model
6969

7070
### Required resources
7171

72-
Before getting started, you need to create a persistent storage layer where the idempotency utility can store its state - your lambda functions will need read and write access to it.
72+
To start, you'll need:
7373

74-
We currently support Amazon DynamoDB and Redis as a storage layer. The following example demonstrates how to create a table in DynamoDB. If you prefer to use Redis, refer go to the section [RedisPersistenceLayer](#redispersistencelayer) section.
74+
1. A persistent storage layer (DynamoDB or [Redis](#redis-as-persistent-storage-layer-provider))
75+
2. An AWS Lambda function with [permissions](#iam-permissions) to use your persistent storage layer
7576

76-
**Default table configuration**
77+
#### DynamoDB table
7778

78-
If you're not [changing the default configuration for the DynamoDB persistence layer](#dynamodbpersistencelayer), this is the expected default configuration:
79+
!!! tip "You can share a single state table for all functions"
7980

80-
| Configuration | Value | Notes |
81-
| ------------------ | ------------ | ----------------------------------------------------------------------------------- |
82-
| Partition key | `id` | |
83-
| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console |
81+
Unless you're looking to use an [existing table or customize each attribute](#dynamodbpersistencelayer), you only need the following:
8482

85-
???+ tip "Tip: You can share a single state table for all functions"
86-
You can reuse the same DynamoDB table to store idempotency state. We add `module_name` and [qualified name for classes and functions](https://peps.python.org/pep-3155/){target="_blank" rel="nofollow"} in addition to the idempotency key as a hash key.
83+
| Configuration | Value | Notes |
84+
| ------------------ | ------------ | -------------------------------------------------------------------------------------------------------- |
85+
| Partition key | `id` | Primary key looks like: <br> `{lambda_fn_name}.{module_name}.{fn_qualified_name}#{idempotency_key_hash}` |
86+
| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console |
87+
88+
Note that `fn_qualified_name` means the [qualified name for classes and functions](https://peps.python.org/pep-3155/){target="_blank" rel="nofollow"} defined in PEP-3155.
89+
90+
##### IaC examples
8791

8892
=== "AWS Serverless Application Model (SAM) example"
8993

@@ -102,19 +106,21 @@ If you're not [changing the default configuration for the DynamoDB persistence l
102106
--8<-- "examples/idempotency/templates/terraform.tf"
103107
```
104108

105-
???+ warning "Warning: Large responses with DynamoDB persistence layer"
106-
When using this utility with DynamoDB, your function's responses must be [smaller than 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items){target="_blank"}.
109+
##### Limitations
110+
111+
* **DynamoDB restricts [item sizes to 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items){target="_blank"}**. This means that if your annotated function's response must be smaller than 400KB, otherwise your function will fail. Consider [Redis](#redis-as-persistent-storage-layer-provider) as an alternative.
112+
113+
* **Expect 2 WCU per non-idempotent call**. During the first invocation, we use `PutItem` for locking and `UpdateItem` for completion. Consider reviewing [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank"}) to estimate cost.
107114

108-
Larger items cannot be written to DynamoDB and will cause exceptions. If your response exceeds 400kb, consider using Redis as your persistence layer.
115+
* **Old boto3 versions can increase costs**. For cost optimization, we use a conditional `PutItem` to always lock a new idempotency record. If locking fails, it means we already have an idempotency record saving us an additional `GetItem` call. However, this is only supported in boto3 `1.26.194` and higher _([June 30th 2023](https://aws.amazon.com/about-aws/whats-new/2023/06/amazon-dynamodb-cost-failed-conditional-writes/){target="_blank"})_.
109116

110-
<!-- markdownlint-disable MD013 -->
111-
???+ info "Info: DynamoDB"
117+
#### Redis cluster
112118

113-
During the first invocation with a payload, the Lambda function executes both a `PutItem` and an `UpdateItem` operations to store the data in DynamoDB. If the result returned by your Lambda is less than 1kb, you can expect 2 WCUs per Lambda invocation.
119+
**TODO**: Experiment bringing upfront Redis even at the cost of readability, as setup and usage are disconnected today causing further harm.
114120

115-
On subsequent invocations with the same payload, you can expect just 1 `PutItem` request to DynamoDB.
121+
##### Constraints
116122

117-
**Note:** While we try to minimize requests to DynamoDB to 1 per invocation, if your boto3 version is lower than `1.26.194`, you may experience 2 requests in every invocation. Ensure to check your boto3 version and review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank"} to estimate the cost.
123+
If you'd like to use Redis, please [read here](#redis-as-persistent-storage-layer-provider) on how to setup and access secrets/SSL certs.
118124

119125
<!-- markdownlint-enable MD013 -->
120126
### Idempotent decorator

0 commit comments

Comments
 (0)