Feature request: Start Evaluating Pydantic v2

[djfurman]

Use case

Pydantic is already included in aws-lamdba-powertools as the provider for deep data validation and parsing as an optional dependency. Pydantic V2 is now in beta evaluation, when it becomes a stable version, it should be introduced as an option into powertools without major breaking changes. We need a way to test the integration with the rest of the library and benchmark performance within Lambda.

Solution/User Experience

Today the parser is included as an extra using Pydantic. With the launch of PydanticV2 we could include a second extra, add max version, or identify if we can do this transparently to the user with an optional switch.

To keep powertools lean, having both dependencies on a single extra seems less than ideal.

What is the best way to evaluate an impact like this without falling out of sync with the enhancements to powertools?

Alternative solutions

- Have two extras tagged to Pydantic versions
- Freeze the parser to Pydantic v1
- Identify if there are any breaking implementations within parser for breaking API changes of the Pydantic upgrade (not personally recommended)
- Identify how to handle the version difference in the full install layer as users will choose one version, not both in their function.

Acknowledgment

• This feature request meets Powertools for AWS Lambda (Python) Tenets
• Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

### Tasks

[djfurman]

Pydantic does not provide a Java or .NET library, nor am I aware of one for TypeScript.

[heitorlessa]

Hey Daniel, good idea!! Have they settled on major breaking changes for Pydantic V2? If not, any major ones that stands out already? We knew v2 was in the works back then so we kept our usage to the bare minimum hoping the transition would be seamless. I suspect the only one surprise might be the coercion mode they changed now (strict vs coerce). Happy with a new optional dep for the time being until we figure this out. Layer will be tricky, it’d require a major version unless the transition is painless — we’d need plenty of guidance like our upgrade guide either way. Thanks for kicking the tires ;)

…

On Thu, 8 Jun 2023 at 20:59, Daniel Furman ***@***.***> wrote: Pydantic does not provide a Java or .NET library, nor am I aware of one for TypeScript. — Reply to this email directly, view it on GitHub <#2427 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBFUP75VIHVMDNFPXCLXKIOIJANCNFSM6AAAAAAY7V5Q2M> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[half2me]

Pydantic 2 has now been released

[leandrodamascena]

Hey everyone! By this month's end, we will create an RFC to discuss the migration from Pydantic v1 to v2. In addition to the breaking changes this migration will cause, we want to understand if there is a way to keep both versions of Pydantic in Powertools and can let customers choose which one they want to use.

Once we have the RFC, I will update this issue.

[half2me]

@leandrodamascena I use both powertools and pydantic in my lambda. Is there some way for me to use pydantic 2 while also letting powertools continue to use v1? If I try to update it, it complains that powertools is pinning the older version

[leandrodamascena]

@leandrodamascena I use both powertools and pydantic in my lambda. Is there some way for me to use pydantic 2 while also letting powertools continue to use v1? If I try to update it, it complains that powertools is pinning the older version

I don't think you can @half2me! Because we still don't support Pydantic v2 and if you try to use Powertools with this version you will get errors. We are prioritizing efforts to support Pydantic v2 as soon as possible.

[leandrodamascena]

Hey everyone! I started working with Pydantic v2 and Powertools and I could use SqsModel with Pydantic v2. I need to dig deeper and investigate this migration further, especially as we want to support v1 and v2 at the same time to avoid breaking changes.

Things I've discovered so far:

1 - All optional fields must have a default value of None otherwise Pydantic will fail.

Pydantic v1

class SqsAttributesModel(BaseModel):
    ApproximateReceiveCount: str
    ApproximateFirstReceiveTimestamp: datetime
    MessageDeduplicationId: Optional[str]
    MessageGroupId: Optional[str]
    SenderId: str
    SentTimestamp: datetime
    SequenceNumber: Optional[str]
    AWSTraceHeader: Optional[str]

Pydantic v2

class SqsAttributesModel(BaseModel):
    ApproximateReceiveCount: str
    ApproximateFirstReceiveTimestamp: datetime
    MessageDeduplicationId: Optional[str] = None
    MessageGroupId: Optional[str] = None
    SenderId: str
    SentTimestamp: datetime
    SequenceNumber: Optional[str] = None
    AWSTraceHeader: Optional[str] = None

2 - The @root_validator decorator is deprecated and will be removed in v3. Even though Pydantic suggests using the new @model_validator decorator, we can still use the @root_validator decorator to avoid breaking changes and plan this removal for Powertools v3.

Pydantic v1

@root_validator(allow_reuse=True)
def check_message_id(cls, values):
    message_id, event_type = values.get("messageId"), values.get("eventType")
    if message_id is not None and event_type != "MESSAGE":
        raise TypeError("messageId is available only when the `eventType` is `MESSAGE`")
    return values

Pydantic v2

@root_validator(allow_reuse=True, skip_on_failure=True)
def check_message_id(cls, values):
    message_id, event_type = values.get("messageId"), values.get("eventType")
    if message_id is not None and event_type != "MESSAGE":
        raise TypeError("messageId is available only when the `eventType` is `MESSAGE`")
    return values

3 - Pydantic package size dropped from ~9MB to ~5.5MB

4 - Let's take this opportunity to check if we can leverage this new version and make some changes to improve the performance in the use of Pydantic.

---

Exampled used:

from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, envelopes
from aws_lambda_powertools.utilities.parser.models import (
    SqsModel,
)

from aws_lambda_powertools import Logger
import pydantic

logger = Logger()

@event_parser(model=SqsModel)
def lambda_handler(event: SqsModel, context):
    logger.info(f"Pydantic version -> {pydantic.__version__}")
    for record in event.Records:
        logger.info(f"Event body -> {record.body}")

Output

❯ sam local invoke --event events/sqs.json
Invoking app.lambda_handler (python3.10)
Local image is up-to-date
Using local image: public.ecr.aws/lambda/python:3.10-rapid-x86_64.

Mounting /home/leandro/DEVEL-PYTHON/tmp/pydantic2/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated, inside runtime container
START RequestId: 8965caed-f0f8-4122-aff0-2d2c69acc631 Version: $LATEST
START RequestId: 8965caed-f0f8-4122-aff0-2d2c69acc631 Version: $LATEST
{"level":"INFO","location":"lambda_handler:16","message":"Pydantic version -> 2.0","timestamp":"2023-07-04 10:31:21,050+0000","service":"service_undefined"}
{"level":"INFO","location":"lambda_handler:18","message":"Event body -> {\"message\": \"foo1\"}","timestamp":"2023-07-04 10:31:21,051+0000","service":"service_undefined"}
{"level":"INFO","location":"lambda_handler:18","message":"Event body -> {\"message\": \"foo1\"}","timestamp":"2023-07-04 10:31:21,051+0000","service":"service_undefined"}
END RequestId: 8965caed-f0f8-4122-aff0-2d2c69acc631
REPORT RequestId: 8965caed-f0f8-4122-aff0-2d2c69acc631	Init Duration: 0.08 ms	Duration: 362.40 ms	Billed Duration: 363 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

Thank you.