WIP feat(python): Legacy Extern #1938

imabhichow · 2025-06-07T08:16:31Z

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

imabhichow · 2025-06-12T23:31:54Z

...ption/runtimes/python/src/aws_dbesdk_dynamodb/internaldafny/extern/InternalLegacyOverride.py

-        # TODO: Implement this
+        # Create and return the legacy override instance
+        legacy_instance = InternalLegacyOverride()
+        legacy_instance.encryptor = legacy_override.encryptor


looking at this, we could actually ignore the encryptor and just call Legacy DDBEC's ItemEncryptor methods encrypt_dynamo_item or encrypt_python_item based on Legacy encryptor(EncryptedClient/EncryptedTable) instance

I'm not totally sure what you mean, but this sounds interesting. If you think it's better, try writing it and I'll take a look

lucasmcdonald3

Most comments are nits/cleanups.

I'm most interested in the other interfaces, so let's get started on that. There may be some fun stuff with how all DBESDK interfaces interact with the EncryptedClient that we should take a look at

lucasmcdonald3 · 2025-06-13T16:34:55Z

DynamoDbEncryption/runtimes/python/pyproject.toml

@@ -13,6 +13,7 @@ include = ["**/internaldafny/generated/*.py"]
 [tool.poetry.dependencies]
 python = "^3.11.0"
 aws-cryptographic-material-providers = { path = "../../../submodules/MaterialProviders/AwsCryptographicMaterialProviders/runtimes/python", develop = false}
+dynamodb_encryption_sdk = "^3.3.0"


DDBEC should be an optional dependency.
Take a look at this page: https://python-poetry.org/docs/pyproject/#extras
You should set it up so that one could install it with a pip command like pip install aws-dbesdk-dynamodb[legacy-ddbec]

lucasmcdonald3 · 2025-06-13T16:56:27Z

...ption/runtimes/python/src/aws_dbesdk_dynamodb/internaldafny/extern/InternalLegacyOverride.py

+            table_name = _dafny.string_of(config.logicalTableName)
+            partition_key_name = _dafny.string_of(config.partitionKeyName)
+            sort_key_name = _dafny.string_of(config.sortKeyName.value) if config.sortKeyName.is_Some else None


Sorry for steering you down the wrong track here, but we shouldn't use _dafny.string_of for anything that handles data.
Its implementation uses a different byte order than the rest of our code.
Check out this dafny-to-native string expression and make a method for it in this file: https://github.com/aws/aws-database-encryption-sdk-dynamodb/blob/python-poc/DynamoDbEncryption/runtimes/python/src/aws_dbesdk_dynamodb/smithygenerated/aws_cryptography_dbencryptionsdk_dynamodb_itemencryptor/dafny_to_smithy.py#L146-L148
I'd replace all instances of _dafny.string_of unless you know it will never handle "special" (non-BMP) characters

lucasmcdonald3 · 2025-06-13T16:57:56Z

...ption/runtimes/python/src/aws_dbesdk_dynamodb/internaldafny/extern/InternalLegacyOverride.py

-    def EncryptItem(input):
-        # TODO: Implement this
-        return Wrappers.Result_Failure("TODO-legacy-Encryptitem")
+            # Map the action from the config to legacy actions


Indentation in this file from here down is odd compared to above

lucasmcdonald3 · 2025-06-13T17:16:17Z

Examples/runtimes/python/Migration/src/ddbec_to_awsdbe/awsdbe/common.py

+from dynamodb_encryption_sdk.material_providers.aws_kms import AwsKmsCryptographicMaterialsProvider
+
+
+def setup_pure_awsdbe_client(kms_key_id: str, ddb_table_name: str):


We should test this with all the boto3 interfaces: EncryptedTables, EncryptedResources, and EncryptedPaginators, since all of these interfaces accept DynamoDbTableEncryptionConfig.

For EncryptedTables and EncryptedResources in particular, we need to make sure that if someone provides a legacy config with a legacy DDBEC encrypted client, then the items are still transformed correctly.
(I think there could be some hiccups with the PR as-is, depending on how our Dafny code works.)

Do some testing and let me know what you find, we can chat about next steps depending on what happens
(We might also need the InternalLegacyOverride to support the legacy interfaces if the items aren't transformed correctly, but we'll see what happens)

I think EncryptedPaginator should just work though.

This is valuable feedback; I plan on doing some experiments testing all potential combinations (matrix), such as each DB-ESDK Interface with each Legacy DDBEC Interface.

Legacy DDBEC also provides a helpful util that transforms python_dict to ddb_item. If we can utilize that util, we could support all interfaces without relying on the encryptors by just calling the legacy item encryptor methods with all matrix combinations like (DB ESDK EncryptedTable with Legacy DDBEC EncryptedPaginator).

lucasmcdonald3 · 2025-06-13T17:17:10Z

...ption/runtimes/python/src/aws_dbesdk_dynamodb/internaldafny/extern/InternalLegacyOverride.py

        # Precondition: The encryptor MUST be a DynamoDBEncryptor
        if not _HAS_LEGACY_DDBEC:
            return InternalLegacyOverride.CreateBuildFailure(
                InternalLegacyOverride.CreateError("Could not find aws-dynamodb-encryption-python installation")
            )
-        if not isinstance(maybe_encryptor, EncryptedClient):
+        if not isinstance(legacy_override.encryptor, EncryptedClient):


Note that we may need to extend this to support legacy EncryptedTable, EncryptedResource, and EncryptedPaginator, depending on what happens when you test the new interfaces for those.

lucasmcdonald3 · 2025-06-13T18:56:33Z

Examples/runtimes/python/Migration/src/ddbec_to_awsdbe/awsdbe/common.py

+        policy=policy,
+    )
+
+    # 1. Create a Keyring. This Keyring will be responsible for protecting the data keys that protect your data.


numbering is off

Examples/runtimes/python/Migration/src/ddbec_to_awsdbe/awsdbe/migration_step_1.py

lucasmcdonald3 · 2025-06-13T19:06:37Z

Examples/runtimes/python/Migration/src/ddbec_to_awsdbe/awsdbe/migration_step_2.py

+  - Create a pure AWS DBESDK client that can only read items encrypted with AWS DBESDK
+  - Create an AWS DBESDK client with legacy override that can read legacy items
+  - Use the pure client to encrypt new items
+  - Use the client with legacy override to read both legacy and new items
+  - Re-encrypt legacy items with the pure AWS DBESDK client


I think there's some redundancies here -- could you tighten this up?

lucasmcdonald3 · 2025-06-13T19:12:33Z

Examples/runtimes/python/Migration/test/ddbec_to_awsdbe/test_utils.py

@@ -0,0 +1,9 @@
+"""Test constants."""


Add the copyright header

Examples/runtimes/python/pyproject.toml

Co-authored-by: Lucas McDonald <lucasmcdonald3@gmail.com>

imabhichow force-pushed the imabhichow/legacy-extern branch 3 times, most recently from 1b314f4 to 21d22dd Compare June 12, 2025 23:26

imabhichow commented Jun 12, 2025

View reviewed changes

chore(python): DDBEC Legacy Extern

522b0f9

imabhichow force-pushed the imabhichow/legacy-extern branch from 21d22dd to 522b0f9 Compare June 13, 2025 09:27

lucasmcdonald3 reviewed Jun 13, 2025

View reviewed changes

optional dependency

0142f15

Co-authored-by: Lucas McDonald <lucasmcdonald3@gmail.com>

imabhichow force-pushed the imabhichow/legacy-extern branch from e75c12e to 0142f15 Compare June 16, 2025 21:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP feat(python): Legacy Extern #1938

WIP feat(python): Legacy Extern #1938

imabhichow commented Jun 7, 2025

Uh oh!

imabhichow Jun 12, 2025

Uh oh!

lucasmcdonald3 Jun 13, 2025

Uh oh!

lucasmcdonald3 left a comment

Uh oh!

lucasmcdonald3 Jun 13, 2025

Uh oh!

lucasmcdonald3 Jun 13, 2025

Uh oh!

lucasmcdonald3 Jun 13, 2025

Uh oh!

lucasmcdonald3 Jun 13, 2025

Uh oh!

imabhichow Jun 13, 2025

Uh oh!

lucasmcdonald3 Jun 13, 2025

Uh oh!

lucasmcdonald3 Jun 13, 2025

Uh oh!

Uh oh!

lucasmcdonald3 Jun 13, 2025

Uh oh!

lucasmcdonald3 Jun 13, 2025

Uh oh!

Uh oh!

Uh oh!

		from dynamodb_encryption_sdk.material_providers.aws_kms import AwsKmsCryptographicMaterialsProvider


		def setup_pure_awsdbe_client(kms_key_id: str, ddb_table_name: str):

WIP feat(python): Legacy Extern #1938

Are you sure you want to change the base?

WIP feat(python): Legacy Extern #1938

Conversation

imabhichow commented Jun 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucasmcdonald3 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!