RFC: Lambda Powertools for Python v2

[rubenfonseca]

Is this related to an existing feature request or issue?

No response

Which AWS Lambda Powertools utility does this relate to?

Powertools itself

Summary

The Python 3.6 Lambda Runtime was deprecated as of Aug 17th 2022. This follows Python 3.6 End-Of-Life (EOL) reached on December 23, 2021. This means it's not possible to create or update Lambda functions using that runtime.

We should take this opportunity to release a new major version of Powertools, and decide what breaking changes to include.

Use case

Customers need to upgrade their Lambda Functions to one of the supported Python runtimes (at the moment of writing, Python 3.7-3.9).

We could use this major upgrade to introduce some breaking changes on Lambda Powertools that will simplify the code and/or improve support for existing features.

Proposal

We strive to make minimal breaking changes due to timing constraints with Python 3.6 deprecation. In ideal scenarios, we would provide ample time, nightly builds, a Beta for a major version, including a linter to help detect and upgrade from breaking changes.

This means, V2 will focus primarily on the following themes:

• Drop Python 3.6. Follows alignment with Lambda runtime deprecation policy. It also allows us to upgrade development dependencies and documentation niceties we couldn't before due to prolonged Python 3.6 EOL life in Lambda.
• Make all dependencies optional to optimize package size. AWS SDK makes up for over 90% of our package size. We also heard from customers that they want to use additional observability providers, thus wanting X-Ray SDK to be optional.
  • Lambda Layer will include all optional dependencies excluding those already available at Lambda runtime to further optimize cold start. This also means dropping SAR Extras as it becomes redundant.
• Remove deprecated features. Specifically sqs_batch_processor and PartialSQSProcessor in favour of BatchProcessor launched 11 months ago that improved accuracy, security, and speed due to the new Lambda integration.
• Improve correctness. Event Handler (API Gateway) doesn't support multi-header and cookies by default as it requires a breaking change. Similarly, Idempotency and Tracer doesn't use fully qualified names, impacting ABC or Classes that use the exact same method name - changing it would impact billions of transactions in production.

Quick summary

Item	Issue/PR	Status	Code change required
Write What's new for v2
Update upgrade guide	#1623	✅
Remove POWERTOOLS_EVENT_HANDLER_DEBUG env var	#1620	✅
Drop support for python 3.6	#1339	✅
Remove the old batch processing legacy implementation (sqs_batch_processor, PartialSQSProcessor)	#1462	✅	Yes
Event Handler REST - multi-value Headers by default, and cookies support	#1455	✅	tests only
Use fully qualified names for idempotency	#1330	✅
Use fully qualified names for tracer subsegments	#1454	✅
Update AppConfig API in Parameters/Feature flags due to GetConfiguration deprecation	#1506	✅
Deprecate SAR Extras	#1543	✅
Make all runtime dependencies optional	#1164	✅
Update docs on required dependencies (validation, parser)	#1573	✅
Event Handler REST v1 supports trailing slash route by default	#1609	✅
Replace AttributeValue in DynamoDBStreamEvent with deserialized Python values	#1619	✅	Yes
Replace email-validator dependency with str in Parser SES Model	#1608	✅

Launch plan

• un-comment "deploy-prod" from publish_v2_layer.yml
• update docs to include SAR v2 and Layer v2
• review Upgrade Guide
• review What's New
• remove V2 banner from docs
• remove V2 admonition from docs

Out of scope

We should make the breaking release as small as possible. For this we should not include in v2:

• modularization to further reduce sizes
• Add a default unit for the metrics utility Provide Default Values for .add_metric(unit=) and .add_metric(value=) #1180

Potential challenges

We need to decide what to do with the Lambda Powertools Layer. One option would be to release a new "V2" layer.

UPDATE: We're launching a separate V2 layer and ARM support along with it!

Dependencies and Integrations

No response

Alternative solutions

No response

Acknowledgment

• This feature request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.

[heitorlessa]

Thank you @rubenfonseca! Let's make Event Handler Headers part of v2 as this will impact our ability to support cookies and multi-value headers without a maintenance hit -- #1455

We should also add that we're going to make minimal breaking changes due to not having ample time to discuss and run experiments with customers - we shall leave those to 3.0 (modularization, potentially making pydantic the default, etc.)

[rubenfonseca]

@heitorlessa thanks, updated! Do you know if an existing issue about removing the first batch processing implementation?

[heitorlessa]

Nope, haven't created yet. We'd need to make a PR to add a deprecation warning too.

https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/aws_lambda_powertools/utilities/batch/sqs.py#L24