You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/utilities/idempotency.md
+24-18Lines changed: 24 additions & 18 deletions
Original file line number
Diff line number
Diff line change
@@ -69,21 +69,25 @@ We provide Infrastrucure as Code examples with [AWS Serverless Application Model
69
69
70
70
### Required resources
71
71
72
-
Before getting started, you need to create a persistent storage layer where the idempotency utility can store its state - your lambda functions will need read and write access to it.
72
+
To start, you'll need:
73
73
74
-
We currently support Amazon DynamoDB and Redis as a storage layer. The following example demonstrates how to create a table in DynamoDB. If you prefer to use Redis, refer go to the section [RedisPersistenceLayer](#redispersistencelayer) section.
74
+
1. A persistent storage layer (DynamoDB or [Redis](#redis-as-persistent-storage-layer-provider))
75
+
2. An AWS Lambda function with [permissions](#iam-permissions) to use your persistent storage layer
75
76
76
-
**Default table configuration**
77
+
#### DynamoDB table
77
78
78
-
If you're not [changing the default configuration for the DynamoDB persistence layer](#dynamodbpersistencelayer), this is the expected default configuration:
79
+
!!! tip "You can share a single state table for all functions"
| TTL attribute name |`expiration`| This can only be configured after your table is created if you're using AWS Console |
81
+
Unless you're looking to use an [existing table or customize each attribute](#dynamodbpersistencelayer), you only need the following:
84
82
85
-
???+ tip "Tip: You can share a single state table for all functions"
86
-
You can reuse the same DynamoDB table to store idempotency state. We add `module_name` and [qualified name for classes and functions](https://peps.python.org/pep-3155/){target="_blank" rel="nofollow"} in addition to the idempotency key as a hash key.
| TTL attribute name |`expiration`| This can only be configured after your table is created if you're using AWS Console |
87
+
88
+
Note that `fn_qualified_name` means the [qualified name for classes and functions](https://peps.python.org/pep-3155/){target="_blank" rel="nofollow"} defined in PEP-3155.
89
+
90
+
##### IaC examples
87
91
88
92
=== "AWS Serverless Application Model (SAM) example"
89
93
@@ -102,19 +106,21 @@ If you're not [changing the default configuration for the DynamoDB persistence l
???+ warning "Warning: Large responses with DynamoDB persistence layer"
106
-
When using this utility with DynamoDB, your function's responses must be [smaller than 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items){target="_blank"}.
109
+
##### Limitations
110
+
111
+
***DynamoDB restricts [item sizes to 400KB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Limits.html#limits-items){target="_blank"}**. This means that if your annotated function's response must be smaller than 400KB, otherwise your function will fail. Consider [Redis](#redis-as-persistent-storage-layer-provider) as an alternative.
112
+
113
+
***Expect 2 WCU per non-idempotent call**. During the first invocation, we use `PutItem` for locking and `UpdateItem` for completion. Consider reviewing [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank"}) to estimate cost.
107
114
108
-
Larger items cannot be written to DynamoDB and will cause exceptions. If your response exceeds 400kb, consider using Redis as your persistence layer.
115
+
***Old boto3 versions can increase costs**. For cost optimization, we use a conditional `PutItem`to always lock a new idempotency record. If locking fails, it means we already have an idempotency record saving us an additional `GetItem` call. However, this is only supported in boto3 `1.26.194` and higher _([June 30th 2023](https://aws.amazon.com/about-aws/whats-new/2023/06/amazon-dynamodb-cost-failed-conditional-writes/){target="_blank"})_.
109
116
110
-
<!-- markdownlint-disable MD013 -->
111
-
???+ info "Info: DynamoDB"
117
+
#### Redis cluster
112
118
113
-
During the first invocation with a payload, the Lambda function executes both a `PutItem` and an `UpdateItem` operations to store the data in DynamoDB. If the result returned by your Lambda is less than 1kb, you can expect 2 WCUs per Lambda invocation.
119
+
**TODO**: Experiment bringing upfront Redis even at the cost of readability, as setup and usage are disconnected today causing further harm.
114
120
115
-
On subsequent invocations with the same payload, you can expect just 1 `PutItem` request to DynamoDB.
121
+
##### Constraints
116
122
117
-
**Note:** While we try to minimize requests to DynamoDB to 1 per invocation, if your boto3 version is lower than `1.26.194`, you may experience 2 requests in every invocation. Ensure to check your boto3 version and review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank"} to estimate the cost.
123
+
If you'd like to use Redis, please [read here](#redis-as-persistent-storage-layer-provider) on how to setup and access secrets/SSL certs.
0 commit comments