RFC: Sensitive data masking utility

[seshubaws]

Is this related to an existing feature request or issue?

#1173

Which AWS Lambda Powertools utility does this relate to?

Other

Summary

Customers would like to obfuscate incoming data for known fields that contain PII, so that they're not passed downstream or accidentally logged. With the increase of batch processing utilities and GDPR, this is one of the hardest tasks for customers today, specially when considering multi-account users.

AWS Encryption SDK is a good starting point but it can be too complex for the average developer, data engineer, or DevOps persona to use. As such, it is highly requested that the Powertools library have a utility to easily mask and/or encrypt sensitive data.

Use case

The use case for this utility would be for developers who want to mask or encrypt sensitive data such as names, addresses, SSNs, etc. in order for them to not be logged in CloudWatch so such data is not compromised, and so that downstream systems like S3, DynamoDB, RDBMS, etc. will not need any additional work on handling PII data.

Additionally, developers should be able to recover encrypted sensitive data to its original form so that they can handle sensitive requests around that data on a as-needed basis.

Proposal

The data masking utility should allow users to mask data, or encrypt and decrypt it. If they would like to encrypt their data, customers should be able to decide for themselves which encryption provider they want to use, though we will provide an out-of-the-box integration with the AWS Encryption SDK. The below code snippet is a rudimentary look at how this utility can be used and how it will function.

Usage

from aws_lambda_powertools.utilities.data_masking.constants import KMS_KEY_ARN
from aws_lambda_powertools.utilities.data_masking import DataMasking
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import AwsEncryptionSdkProvider


def lambda_handler():
    
   data_masker = DataMasking()
    data = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "johndoe@example.com",
               "address": {
                    "street": "123 Main St", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }

    masked = data_masker.mask(data=data, fields=["email", "address.street"])
    """
    masked = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "*****",
               "address": {
                    "street": "*****", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }
   """

    encryption_provider = AwsEncryptionSdkProvider(keys=[KMS_KEY_ARN])
    data_masker = DataMasking(provider=encryption_provider)

    encrypted = data_masker.encrypt(data=data, fields=["email", "address.street"])
    """
    encrypted = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "InRoaXMgaXMgYSBzdHJpbmciHsLZGx2na-XzP_TB5Bf2LNU1bLc",
               "address": {
                    "street": "XMgYSB_KDddaDJYMb-JpbmGnagTklwQ-msdaDLP", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }
   """
 
    decrypted = data_masker.decrypt(data=encrypted, fields=["email", "address.street"])
    """
    data = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "johndoe@example.com",
               "address": {
                    "street": "123 Main St", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }
   """

AWS Encryption SDK

The AWS Encryption SDK is a client-side encryption library that makes it easier to encrypt and decrypt data of any type in your application. The Encryption SDK is available in all the languages that Powertools supports. You can use it with customer master keys in AWS Key Management Service (AWS KMS), though the library does not require any AWS service. When you encrypt data, the SDK returns a single, portable encrypted message that includes the encrypted data and encrypted data keys. This object is designed to work in many different types of applications. You can specify many of the encryption options, including selecting an encryption and signing algorithm.

Latencies
Latencies for using this utility with the AWS Encryption SDK in Lambda functions configured with 128MB, 1024MB, and 1769MB, respectively.

plugins.metrics-by-endpoint.response_time./Prod/function128:
  min: ......................................................................... 0
  max: ......................................................................... 5373
  median: ...................................................................... 561.2
  p95: ......................................................................... 713.5
  p99: ......................................................................... 1620
plugins.metrics-by-endpoint.response_time./Prod/function1024:
  min: ......................................................................... 0
  max: ......................................................................... 5039
  median: ...................................................................... 89.1
  p95: ......................................................................... 144
  p99: ......................................................................... 232.8
plugins.metrics-by-endpoint.response_time./Prod/function1769:
  min: ......................................................................... 0
  max: ......................................................................... 1726
  median: ...................................................................... 82.3
  p95: ......................................................................... 133
  p99: ......................................................................... 183.1

Custom encryption
If customer would like to use another encryption provider, or define their own encrypt and decrypt functions, we will define an interface that the customer can implement and pass in to the DataMaskingUtility class.

from aws_lambda_powertools.utilities.data_masking.provider import BaseProvider
from itsdangerous.url_safe import URLSafeSerializer

class MyCustomEncryption(BaseProvider):
    def __init__(self, secret):
        self.secret = URLSafeSerializer(secret)

    def encrypt(self, value: str) -> str:
        if value is None:
            return value
        return self.secret.dumps(value)

    def decrypt(self, value: str) -> str:
        if value is None:
            return value
        return self.secret.loads(value)


def lambda_handler():
    data = {
        "id": 1,
        "name": "John Doe",
        "age": 30,
        "email": "johndoe@example.com",
        "address": {
            "street": "123 Main St", 
            "city": "Anytown", 
            "state": "CA", 
            "zip": "12345"
        },
    }

    masking_provider = MyCustomEncryption(secret="secret-key")
    data_masker = DataMasking(provider=masking_provider)

    encrypted = data_masker.encrypt(data, fields=["email", "address.street"])
    """
    encrypted = {
        "id": 1,
        "name": "John Doe",
        "age": 30,
        "email": "InRoaXMgaXMgYSBzdHJpbmciHsLZGx2na-XzP_TB5Bf2LNU1bLc",
        "address": {
            "street": "XMgYSB_KDddaDJYMb-JpbmGnagTklwQ-msdaDLP", 
            "city": "Anytown", 
            "state": "CA", 
            "zip": "12345"
        },
    }
   """

   decrypted = data_masker.decrypt(data=encrypted, fields=["email", "address.street"])
    """
    decrypted = {
              "id": 1,
               "name": "John Doe",
               "age": 30,
               "email": "johndoe@example.com",
               "address": {
                    "street": "123 Main St", 
                    "city": "Anytown", 
                    "state": "CA", 
                    "zip": "12345"
               },
      }
   """

Out of scope

Traversing an arbitrary dictionary will be out of scope for the initial launch of this tool. This feature will be to receive instructions as to where in the given dictionary it should mask/unmask the data.

We still need to determine the most efficient method of taking input JSON path masking or encrypting the value at that path. JMESPath or JSON path can be considered for simple use cases but we need to find the fastest method.

Potential challenges

• We will need to discuss the design and implementation with a security expert to ensure safety.
• Also need to determine if we should support envelope encryption for customers using the AWS Encryption SDK
  • Envelope encryption is the practice of encrypting plaintext data with a data key, and then encrypting the data key under another key. But, eventually, one key must remain in plaintext so you can decrypt the keys and your data. This top-level plaintext key encryption key is known as the root key. You can store root keys in AWS KMS, known as AWS KMS keys.
• Need to decide what the best name for the methods are. Should they still be called encrypt and decrypt even in the case where the customer only masks and won't be able to decrypt later? Should we have a mask method in the interface that is always accessible so that users can irretrievably mask some data and also encrypt some data?

Dependencies and Integrations

Integration with the AWS Encryption SDK.

Alternative solutions

No response

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[heitorlessa]

Prioritising for review tomorrow - missed our typical response time due to team offsite.

We'll revert with comments by tomorrow COB

[rubenfonseca]

Hi @seshubaws thank you for your great work on the RFC. I especially liked the way you already pre-tested some solutions to find some bottlenecks.

I have two concerns:

• it seems to me the "JSON path" method to identify fields that can contain PII is very limiting, compared to the offer we have on CloudWatch right now. With CloudWatch you instead declare categories of PII you are interested in masking, and AWS will mask automatically no matter where they appear. I believe Heitor's proposal where we use pydantic's parser could be a little better, but still do you think we can inverse the responsibilities here, and help the user declare "what to mask" vs "where to mask"?

• regarding ItsDangerous, I understand the benefit of having something simpler. However, handling private keys and secrets on our own seems scary to me, and I would much prefer to build on top of the existing SDK and mechanisms, even if it means it will be slower. Alternatively, ItsDangerous could probably be used as long as we leave all the secret management to KMS.

Last, one thing I would love to see, is to augment objects that have PII with the corresponding encrypted version. Example:

{ "email": "foo@bar.com", name: "John" }

If we are interested in masking emails, this could turn into

{ "email": "*** 1 ***", name: "John", "*** 1 ***": "encrypted/signed version of the PII" }

This way, anyone with access to the decryption keys would be able to just augment the object with the original data, instead of having to find it on external systems.

[heitorlessa]

This is great, I'm so excited we're moving along with this now. A few immediate comments:

• We should strive for cross-language compatibility. It's not a hard requirement, but it'd be great if ~50-70% is achievable to aid feature parity in the future. Other Powertools languages will follow the docs and features, it doesn't have to be a 1:1 mapping, otherwise we risk becoming half-good everywhere - we delight customers in their language of choice.
  • Q: For AST, are you using stdlib ast or a 3rd party library? If the latter, what's their license and package size?
• Batteries included but swappable. I agree with Ruben that AWS Encryption SDK however heavy it addresses common security concerns we are not typically aware. We should be mindful to support 80% of customers looking for correctness and security while also allowing 20% of customers to bring their own implementation (e.g. ItsDangerous is everywhere) without bloating their dependencies with our recommended choice. For example, ItsDangerous is roughly 36K zipped while AWS Encryption SDK brings 18M zipped, largely due to AWS SDK and other heavy hitters. Encryption SDK alone might make it difficult for some Data Engineers/Scientists to use it, hence having a Provider helps us meet them where they are.
  • Q: Could you include what the experience would look like to bring-your-own-masking provider?
  • Q: Based on your research with both Encryption SDK and ItsDangerous, what common nomenclature could we use for interfaces? e.g., sign, dumps, etc.
• What vs Where to mask. Unless Ruben has suggestions on how to address it for customers not using Parser (SRE, DevOps, Data Scientists), we'd want to support - however, it doesn't have to be at launch. [What] We could start with the Pydantic above and focus on getting a Provider interface, and correctness approach first. [Where] Once that's out of the picture, we can hear from customers on how they'd plan to use the JSON Path-like to scrub data recursively.
  • NOTE: They should be easily composable, since one traverses and one mask. If one doesn't have a dict data structure, you simply use a Provider to mask data.
• Value of CloudWatch Logs PII. We recently announced native support for masking sensitive data in CloudWatch Logs, quite similarly to SNS. Part of Powertools exists because of integration and feature gaps between services, however, we should not assume that everyone using Lambda will want their logs dumped in CloudWatch - see Observability providers top demanded feature. Instead, we should recognize the plurality of customers we have. Depending on volume and need, it might be more economically viable to use this feature vs a managed service.
  • NOTE: Alternatively, if a customer wants to mask PII before they send it to downstream services via REST/GraphQL APIs, or non-SNS messaging services (EventBridge, Kinesis, SQS, Kafka, MQ), this is still a value add for them to rely on this feature. Over time, we will continue to voice our customers feedback to make it into a native feature to reduce operational overhead - until then, I'd be happy to hear otherwise why not.

On UX:

• A draft proof of concept would be great to improve this discussion.
• We'd want to avoid high-level top imports to prevent cold starts, like what happened to Tracer and thus requiring additional logic for lazy imports. Instead, use aws_lambda_powertools.utilities.data_masking.
• Let's not mutate implicitly and always return the data masked, despite Python using pass-by-reference. This makes it easier for cross-languages too.
• Prefer data parameter over event parameter. This enables a near future where we officially support this feature beyond Lambda runtimes - this feature could easily be used anywhere.
• Customers have more than a Dict, so let's account for those too. This makes it easier to interop with existing systems they might have too to ease migration (on/off), if necessary - e.g., Signer.sign("my_email@domain.com")

Super looking forward to this.

Thank you for all the hard work you already put into this.

[lpsuerj]

I'm excited about all the possibilities we can uncover with this feature on the security front! Few things that come to my mind about it:

1. When logs are not configured to be sent to CloudWatch logs, we need to ensure there's a way to show evidence of security control implementation effectiveness. Encryption of sensitive data is a high priority for security audit teams and to enabled an easy path to comply with regulator requirements (internal and external) would be a big win.

2. AWS is broadly adopting a new security industry-wide logging format called OCSF. I't be great if we could embrace the same format for a future potential integration with services like AWS Security Lake for encryption events logs. Ideally we should still log the encryption activities following defined patterns in the Encryption SDK. This is NOT a hard requirement but a great nice to have

I'm looking forward to deep dive in this topic and join the team bringing this to life!

[ran-isenberg]

Hey, really liking this idea!
I'd like to share my two cents as this is an issue I also deal with in my day to day.
I'd usually use a KMS CMK and boto to encrypt PII and in many cases either the customer brings its own key or i have a key per tenant but the encryption itself is required to be FIPS certified and other goodies.
For safe logging, if you parse your input with the Parser utility you can define any sensitive information as SecretStr (https://docs.pydantic.dev/usage/types/#secret-types) which causes it to printed as '****' when serialized to a string in the logger.

[seshubaws]

@rubenfonseca Thank you for your comments, let me know if I haven't answered all your questions!

It seems to me the “JSON path” method to identify fields that can contain PII is very limiting, compared to the offer we have on CloudWatch right now. With CloudWatch you instead declare categories of PII you are interested in masking, and AWS will mask automatically no matter where they appear. I believe Heitor’s proposal where we use pydantic’s parser could be a little better, but still do you think we can inverse the responsibilities here, and help the user declare “what to mask” vs “where to mask”?

Yes, that makes sense to me! I played around a little with CW’s new data masking tool and it seems they might be using regex for pattern matching for the desired categories users want to mask, but not exactly sure (let me know if you know what how they implemented this). I’m not very familiar with Pydantic but if it will help users identify more generic PII fields, we should definitely use that.

regarding ItsDangerous, I understand the benefit of having something simpler. However, handling private keys and secrets on our own seems scary to me, and I would much prefer to build on top of the existing SDK and mechanisms, even if it means it will be slower. Alternatively, ItsDangerous could probably be used as long as we leave all the secret management to KMS.

Yes, I think this goes along with what Heitor mentioned about designing a bring-your-own-masking-provider type interface. As both the AWS Encryption SDK and ItsDangerous have their tradeoffs, we should leave it up to the customer to decide which would fit best with their specific use case. I’ll work on coming up with a UX draft for this.

Last, one thing I would love to see, is to augment objects that have PII with the corresponding encrypted version. Example:

We could definitely do this! I am a little confused on what use case would require a field to be masked and also encrypted though, as I am under the impression that if it’s encrypted, then it’s just as good as being masked with the added benefit of only users with access can decrypt it.