From 4d9b2a98d16ed1ba092e072392c6eaaf0f80e5e2 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Wed, 14 Oct 2020 20:40:43 +0200 Subject: [PATCH 01/26] docs: initial sketch of parser docs --- docs/content/utilities/parser.mdx | 53 +++++++++++++++++++++++++++++++ docs/gatsby-config.js | 3 +- 2 files changed, 55 insertions(+), 1 deletion(-) create mode 100644 docs/content/utilities/parser.mdx diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx new file mode 100644 index 00000000000..3b4d7471c72 --- /dev/null +++ b/docs/content/utilities/parser.mdx @@ -0,0 +1,53 @@ +--- +title: Parser +description: Utility +--- + + +import Note from "../../src/components/Note" + + + It requires an extra dependency before using it. +
+ +This utility provides data parsing and deep validation using [Pydantic](https://pydantic-docs.helpmanual.io/). + +**Key features** + +* Defines data in pure Python classes, then parse, validate and extract only what you want +* Built-in envelopes to unwrap, extend, and validate popular event sources payloads +* Enforces type hints at runtime with user friendly errors + +**Extra dependency** + + + This will install pydantic and typing_extensions +
+ +Install parser's extra dependencies using **`pip install aws-lambda-powertools[pydantic]`**. + +## Defining models + +You can define models to parse incoming events by inheriting from `BaseModel`. + +```python:title=hello_world_model.py +from aws_lambda_powertools.utilities.parser import BaseModel + +class HelloWorldModel(BaseModel): + message: str + +payload = {"message": "hello world"} +parsed_payload = HelloWorldModel(**payload) + +assert parsed_payload.message == payload["message"] +``` + +These are simply Python classes that inherit from BaseModel, and use type hints to instruct **parser** to enforce it at runtime. The advantage here is that they can be [recursive, dumped as JSON, JSON Schema, Dicts, have validation and more](https://pydantic-docs.helpmanual.io/usage/models/). + +You can also even use [a code generator tool](https://github.com/koxudaxi/datamodel-code-generator/) to create models from JSON Schemas, OpenAPI, etc. + +## Parsing events + +You can parse inbound events using **parser** decorator. + +You can also use the standalone **parse** function, if you want more control over data validation process such as handling data that doesn't conform with your model. diff --git a/docs/gatsby-config.js b/docs/gatsby-config.js index 087f23a9634..af82f3e2e3d 100644 --- a/docs/gatsby-config.js +++ b/docs/gatsby-config.js @@ -35,7 +35,8 @@ module.exports = { 'utilities/batch', 'utilities/typing', 'utilities/validation', - 'utilities/data_classes' + 'utilities/data_classes', + 'utilities/parser' ], }, navConfig: { From 190703b14255b09a4759c6169fcbdb884664e926 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Wed, 14 Oct 2020 21:36:15 +0200 Subject: [PATCH 02/26] docs: initial structure for parser docs --- docs/content/utilities/parser.mdx | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 3b4d7471c72..faf89a2fc55 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -48,6 +48,28 @@ You can also even use [a code generator tool](https://github.com/koxudaxi/datamo ## Parsing events -You can parse inbound events using **parser** decorator. +You can parse inbound events using **event_parser** decorator. You can also use the standalone **parse** function, if you want more control over data validation process such as handling data that doesn't conform with your model. + +**TBW** + +### event_parser decorator + +**TBW** + +### Parse function + +**TBW** + +## Built-in envelopes + +**TBW** + +## Extending built-in models + +**TBW** + +## Deep model validation + +**TBW** From f0a8a1803ebad33aed410a5cad677bcd6185b404 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Tue, 20 Oct 2020 10:16:19 +0200 Subject: [PATCH 03/26] docs: add 101 parsing events content --- docs/content/utilities/parser.mdx | 57 ++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 13 deletions(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index faf89a2fc55..d19e00519f2 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -36,31 +36,62 @@ from aws_lambda_powertools.utilities.parser import BaseModel class HelloWorldModel(BaseModel): message: str -payload = {"message": "hello world"} -parsed_payload = HelloWorldModel(**payload) - -assert parsed_payload.message == payload["message"] +class NestedHelloWorldModel(BaseModel): + payload: HelloWorldModel ``` -These are simply Python classes that inherit from BaseModel, and use type hints to instruct **parser** to enforce it at runtime. The advantage here is that they can be [recursive, dumped as JSON, JSON Schema, Dicts, have validation and more](https://pydantic-docs.helpmanual.io/usage/models/). +These are simply Python classes that inherit from BaseModel. **Parser** enforces type hints declared in your model at runtime. + + + -You can also even use [a code generator tool](https://github.com/koxudaxi/datamodel-code-generator/) to create models from JSON Schemas, OpenAPI, etc. + + Looking to auto-generate models from JSON, YAML, JSON Schemas, OpenApi, etc? +

+ Use Koudai Aono's data model code generator tool for Pydantic +
## Parsing events -You can parse inbound events using **event_parser** decorator. +You can parse inbound events using **event_parser** decorator, or the standalone `parse` function. -You can also use the standalone **parse** function, if you want more control over data validation process such as handling data that doesn't conform with your model. +### event_parser decorator -**TBW** +Use the decorator for fail fast scenarios where you want your Lambda function to raise an exception in the event of a malformed payload. -### event_parser decorator +`event_parser` decorator will throw a `ModelValidationError` if your event cannot be parsed according to the model. -**TBW** +```python=:title=event_parser_decorator.py +from aws_lambda_powertools.utilities.parser import parse, ModelValidationError +from aws_lambda_powertools.utilities.typing import LambdaContext -### Parse function +@event_parser(model=HelloWorldModel) +def handler(event, context: LambdaContext): + pass -**TBW** +handler(event=payload, context=LambdaContext() +``` + +### parse function + +Use this standalone function when you want more control over the data validation process, for example returning a 400 error for malformed payloads. + +```python:title=parse_standalone_example.py +from aws_lambda_powertools.utilities.parser import parse, ModelValidationError + +payload = {"message": "hello world"} + +def my_function(): + try: + parsed_payload = parse(event=payload, model=HelloWorldModel) # highlight-line + # payload dict is now parsed into our model + return assert parsed_payload.message == payload["message"] + except ModelValidationError: + return { + "status_code": 400, + "message": "Invalid input" + } +``` ## Built-in envelopes From bf294ee22412eeff38241de91fe1d9cd1b6e7c7c Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Tue, 20 Oct 2020 10:17:06 +0200 Subject: [PATCH 04/26] fix: parse high level import --- aws_lambda_powertools/utilities/parser/__init__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/aws_lambda_powertools/utilities/parser/__init__.py b/aws_lambda_powertools/utilities/parser/__init__.py index 62aa4bd73d7..08049922a9c 100644 --- a/aws_lambda_powertools/utilities/parser/__init__.py +++ b/aws_lambda_powertools/utilities/parser/__init__.py @@ -3,11 +3,12 @@ from . import envelopes from .envelopes import BaseEnvelope from .exceptions import ModelValidationError -from .parser import event_parser +from .parser import event_parser, parse from .pydantic import BaseModel, root_validator, validator __all__ = [ "event_parser", + "parse", "envelopes", "BaseEnvelope", "BaseModel", From 96fc4445d78e2761a30cfb2313c52698497f473e Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Tue, 20 Oct 2020 10:18:20 +0200 Subject: [PATCH 05/26] chore: typo on code generation tool --- docs/content/utilities/parser.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index d19e00519f2..28de905a320 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -48,7 +48,7 @@ These are simply Python classes that inherit from BaseModel. **Parser** enforces Looking to auto-generate models from JSON, YAML, JSON Schemas, OpenApi, etc?

- Use Koudai Aono's data model code generator tool for Pydantic + Use Koudai Aono's data model code generation tool for Pydantic
## Parsing events From ba7cd29c4247a51538e7d1061bebe7207e14448d Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Tue, 20 Oct 2020 16:29:25 +0200 Subject: [PATCH 06/26] docs: use non-hello world model to better exemplify parsing Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 48 ++++++++++++++++++++++--------- 1 file changed, 34 insertions(+), 14 deletions(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 28de905a320..3d015674e4d 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -32,12 +32,18 @@ You can define models to parse incoming events by inheriting from `BaseModel`. ```python:title=hello_world_model.py from aws_lambda_powertools.utilities.parser import BaseModel - -class HelloWorldModel(BaseModel): - message: str - -class NestedHelloWorldModel(BaseModel): - payload: HelloWorldModel +from typing import List, Optional + +class OrderItem(BaseModel): + id: int + quantity: int + description: str + +class Order(BaseModel): + id: int + description: str + items: List[OrderItem] # nesting models are supported + optional_field: Optional[str] # this field may or may not be available when parsing ``` These are simply Python classes that inherit from BaseModel. **Parser** enforces type hints declared in your model at runtime. @@ -65,11 +71,14 @@ Use the decorator for fail fast scenarios where you want your Lambda function to from aws_lambda_powertools.utilities.parser import parse, ModelValidationError from aws_lambda_powertools.utilities.typing import LambdaContext -@event_parser(model=HelloWorldModel) -def handler(event, context: LambdaContext): - pass +@event_parser(model=Order) # highlight-line +def handler(event: Order, context: LambdaContext): + print(event.id) + print(event.description) + print(event.items) -handler(event=payload, context=LambdaContext() + order_items = [items for item in event.items] + ... ``` ### parse function @@ -79,17 +88,28 @@ Use this standalone function when you want more control over the data validation ```python:title=parse_standalone_example.py from aws_lambda_powertools.utilities.parser import parse, ModelValidationError -payload = {"message": "hello world"} +# Raw event for the Order model we've defined earlier +payload = { + "id": 10876546789, + "description": "My order", + "items": [ + { + "id": 1015938732, + "quantity": 1, + "description": "item xpto" + } + ] +} def my_function(): try: - parsed_payload = parse(event=payload, model=HelloWorldModel) # highlight-line + parsed_payload: Order = parse(event=payload, model=HelloWorldModel) # highlight-line # payload dict is now parsed into our model - return assert parsed_payload.message == payload["message"] + return parsed_payload.items except ModelValidationError: return { "status_code": 400, - "message": "Invalid input" + "message": "Invalid order" } ``` From cb8d6a00914f82c8ab922b2a8c37de89cbedf29d Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Tue, 20 Oct 2020 16:53:23 +0200 Subject: [PATCH 07/26] fix: ensures parser can take json strings as input --- .../utilities/parser/parser.py | 3 +++ docs/content/utilities/parser.mdx | 23 ++++++++++++++++++- tests/functional/parser/test_parser.py | 13 ++++++++++- 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/aws_lambda_powertools/utilities/parser/parser.py b/aws_lambda_powertools/utilities/parser/parser.py index a58ee90f4e9..1f3a2d4ace0 100644 --- a/aws_lambda_powertools/utilities/parser/parser.py +++ b/aws_lambda_powertools/utilities/parser/parser.py @@ -149,6 +149,9 @@ def handler(event: Order, context: LambdaContext): try: logger.debug("Parsing and validating event model; no envelope used") + if isinstance(event, str): + return model.parse_raw(event) + return model.parse_obj(event) except (ValidationError, TypeError) as e: raise ModelValidationError("Input event does not conform with model") from e diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 3d015674e4d..376ce03e08c 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -59,7 +59,7 @@ These are simply Python classes that inherit from BaseModel. **Parser** enforces ## Parsing events -You can parse inbound events using **event_parser** decorator, or the standalone `parse` function. +You can parse inbound events using **event_parser** decorator, or the standalone `parse` function. Both are also able to parse either dictionary or JSON string as an input. ### event_parser decorator @@ -70,6 +70,20 @@ Use the decorator for fail fast scenarios where you want your Lambda function to ```python=:title=event_parser_decorator.py from aws_lambda_powertools.utilities.parser import parse, ModelValidationError from aws_lambda_powertools.utilities.typing import LambdaContext +import json + +# Raw event for the Order model we've defined earlier +payload = { + "id": 10876546789, + "description": "My order", + "items": [ + { + "id": 1015938732, + "quantity": 1, + "description": "item xpto" + } + ] +} @event_parser(model=Order) # highlight-line def handler(event: Order, context: LambdaContext): @@ -79,6 +93,9 @@ def handler(event: Order, context: LambdaContext): order_items = [items for item in event.items] ... + +handler(event=payload, context=LambdaContext()) +handler(event=json.dumps(payload), context=LambdaContext()) # also works if event is a JSON string ``` ### parse function @@ -113,6 +130,10 @@ def my_function(): } ``` +### Error handling + +**TBW** + ## Built-in envelopes **TBW** diff --git a/tests/functional/parser/test_parser.py b/tests/functional/parser/test_parser.py index 162b52ee439..ec802471585 100644 --- a/tests/functional/parser/test_parser.py +++ b/tests/functional/parser/test_parser.py @@ -1,4 +1,5 @@ -from typing import Dict +import json +from typing import Dict, Union import pytest @@ -55,3 +56,13 @@ def handle_no_envelope(event: Dict, _: LambdaContext): with pytest.raises(exceptions.InvalidModelTypeError): handle_no_envelope(event=dummy_event, context=LambdaContext()) + + +def test_parser_event_as_json_string(dummy_event, dummy_schema): + dummy_event = json.dumps(dummy_event["payload"]) + + @event_parser(model=dummy_schema) + def handle_no_envelope(event: Union[Dict, str], _: LambdaContext): + return event + + handle_no_envelope(dummy_event, LambdaContext()) From 0ed746b6481a17716e5a95b893dce77b695daecf Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Tue, 20 Oct 2020 18:08:56 +0200 Subject: [PATCH 08/26] docs: add data model validation section Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 100 ++++++++++++++++++++++++++++-- 1 file changed, 94 insertions(+), 6 deletions(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 376ce03e08c..a22c9bb1fa5 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -130,18 +130,106 @@ def my_function(): } ``` -### Error handling +### Data model validation -**TBW** + + This is radically different from the Validator utility which validates events against JSON Schema. +
-## Built-in envelopes +You can use parser's validator for deep inspection of object values and complex relationships. -**TBW** +There are two types of class method decorators you can use: -## Extending built-in models +* **`validator`** - Useful to quickly validate an individual field and its value +* **`root_validator`** - Useful to validate the entire model's data + +Keep the following in mind regardless of which decorator you end up using it: + +* You must raise either `ValueError`, `TypeError`, or `AssertionError` when value is not compliant +* You must return the value(s) itself if compliant + +#### Validating fields + +Quick validation to verify whether the field `message` has the value of `hello world`. + +```python:title=deep_data_validation.py +from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator + +class HelloWorldModel(BaseModel): + message: str + + @validator('message') # highlight-line + def is_hello_world(cls, v): + if v != "hello world": + raise ValueError("Message must be hello world!") + return v + +parse(model=HelloWorldModel, event={"message": "hello universe"}) +``` + +If you run as-is, you should expect the following error with the message we provided in our exception: + +``` +message + Message must be hello world! (type=value_error) +``` + +Alternatively, you can pass `'*'` as an argument for the decorator so that you can validate every value available. + +```python:title=validate_all_field_values.py +from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator + +class HelloWorldModel(BaseModel): + message: str + sender: str + + @validator('*') # highlight-line + def has_whitespace(cls, v): + if ' ' not in v: + raise ValueError("Must have whitespace...") + + return v + +parse(model=HelloWorldModel, event={"message": "hello universe", "sender": "universe"}) +``` + +#### Validating entire model + +`root_validator` can help when you have a complex validation mechanism. For example finding whether data has been omitted, comparing field values, etc. + +```python:title=validate_all_field_values.py +from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator + +class UserModel(BaseModel): + username: str + password1: str + password2: str + + @root_validator + def check_passwords_match(cls, values): + pw1, pw2 = values.get('password1'), values.get('password2') + if pw1 is not None and pw2 is not None and pw1 != pw2: + raise ValueError('passwords do not match') + return values + +payload = { + "username": "universe", + "password1": "myp@ssword", + "password2": "repeat password" +} + +parse(model=UserModel, event=payload) +``` + + + You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in Pydantic's documentation. +
+ + +## Built-in envelopes **TBW** -## Deep model validation +## Extending built-in models **TBW** From 75dc529227025af2f2d95f0c27bf7ddf5a8e53dd Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Thu, 22 Oct 2020 18:43:33 +0200 Subject: [PATCH 09/26] docs: add envelope section --- .../utilities/parser/__init__.py | 3 +- .../utilities/parser/envelopes/__init__.py | 4 +- .../utilities/parser/envelopes/dynamodb.py | 4 +- .../utilities/parser/envelopes/sqs.py | 2 +- docs/content/utilities/parser.mdx | 122 +++++++++++++++++- tests/functional/parser/test_dynamodb.py | 2 +- 6 files changed, 127 insertions(+), 10 deletions(-) diff --git a/aws_lambda_powertools/utilities/parser/__init__.py b/aws_lambda_powertools/utilities/parser/__init__.py index 08049922a9c..a010301e7fb 100644 --- a/aws_lambda_powertools/utilities/parser/__init__.py +++ b/aws_lambda_powertools/utilities/parser/__init__.py @@ -4,7 +4,7 @@ from .envelopes import BaseEnvelope from .exceptions import ModelValidationError from .parser import event_parser, parse -from .pydantic import BaseModel, root_validator, validator +from .pydantic import BaseModel, Field, root_validator, validator __all__ = [ "event_parser", @@ -12,6 +12,7 @@ "envelopes", "BaseEnvelope", "BaseModel", + "Field", "validator", "root_validator", "ModelValidationError", diff --git a/aws_lambda_powertools/utilities/parser/envelopes/__init__.py b/aws_lambda_powertools/utilities/parser/envelopes/__init__.py index 766021a3f92..2398840a756 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/__init__.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/__init__.py @@ -1,6 +1,6 @@ from .base import BaseEnvelope -from .dynamodb import DynamoDBEnvelope +from .dynamodb import DynamoDBStreamEnvelope from .event_bridge import EventBridgeEnvelope from .sqs import SqsEnvelope -__all__ = ["DynamoDBEnvelope", "EventBridgeEnvelope", "SqsEnvelope", "BaseEnvelope"] +__all__ = ["DynamoDBStreamEnvelope", "EventBridgeEnvelope", "SqsEnvelope", "BaseEnvelope"] diff --git a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py index ef166a5c48f..69585e02e15 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py @@ -10,7 +10,7 @@ logger = logging.getLogger(__name__) -class DynamoDBEnvelope(BaseEnvelope): +class DynamoDBStreamEnvelope(BaseEnvelope): """ DynamoDB Stream Envelope to extract data within NewImage/OldImage Note: Values are the parsed models. Images' values can also be None, and @@ -31,6 +31,8 @@ def parse(self, data: Dict[str, Any], model: BaseModel) -> List[Dict[Literal["Ne ------- List List of records parsed with model provided + + """ parsed_envelope = DynamoDBStreamModel(**data) output = [] diff --git a/aws_lambda_powertools/utilities/parser/envelopes/sqs.py b/aws_lambda_powertools/utilities/parser/envelopes/sqs.py index 7bf326206f3..61d9a4b40c2 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/sqs.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/sqs.py @@ -19,7 +19,7 @@ class SqsEnvelope(BaseEnvelope): all items in the list will be parsed as str and npt as JSON (and vice versa) """ - def parse(self, data: Dict[str, Any], model: Union[BaseModel, str]) -> List[Union[BaseModel, str]]: + def parse(self, data: Dict[str, Any], model: Union[BaseModel, str]) -> List[BaseModel]: """Parses records found with model provided Parameters diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index a22c9bb1fa5..8d8cbf11478 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -57,6 +57,10 @@ These are simply Python classes that inherit from BaseModel. **Parser** enforces Use Koudai Aono's data model code generation tool for Pydantic
+### Extending built-in models + +**TBW** + ## Parsing events You can parse inbound events using **event_parser** decorator, or the standalone `parse` function. Both are also able to parse either dictionary or JSON string as an input. @@ -226,10 +230,120 @@ parse(model=UserModel, event=payload)
-## Built-in envelopes +## Envelopes -**TBW** +Envelope parameter is useful when your actual payload is wrapped around a known structure, for example Lambda Event Sources like Amazon EventBridge. -## Extending built-in models +Example of parsing a model found in an event coming from EventBridge, where all you want is what's inside the `detail` key. -**TBW** +```python:title=parse_eventbridge_payload.py +from aws_lambda_powertools.utilities.parser import parse, BaseModel, envelopes + +class UserModel(BaseModel): + username: str + password1: str + password2: str + +payload = { + "version": "0", + "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", + "detail-type": "CustomerSignedUp", + "source": "CustomerService", + "account": "111122223333", + "time": "2020-10-22T18:43:48Z", + "region": "us-west-1", + "resources": ["some_additional_"], + # highlight-start + "detail": { + "username": "universe", + "password1": "myp@ssword", + "password2": "repeat password" + } + # highlight-end +} + +ret = parse(model=UserModel, envelope=envelopes.EventBridgeModel, payload=payload) # highlight-line + +# Parsed model only contains our actual model, not the entire EventBridge + Payload parsed +assert ret.password1 == ret.password2 +``` + +**What's going on here, you might ask**: + +1. We imported built-in `models` from the parser utility +2. Used `envelopes.EventBridgeModel` as the envelope for our `UserModel` model +3. Parser parsed the original event against the EventBridge model +4 Parser then parsed the `detail` key using `UserModel` + +### Built-in envelopes + +Parser comes with the following built-in envelopes, where `BaseModel` in the return section is your given model. + +Envelope name | Behaviour | Return +------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------ +**DynamoDBStreamEnvelope** | 1. Parses data using `DynamoDBStreamModel`.
2. Parses records in `NewImage` and `OldImage` keys using your model.
3. Returns a list with a dictionary containing `NewImage` and `OldImage` keys | `List[Dict[Literal["NewImage", "OldImage"], BaseModel]]` +**EventBridgeEnvelope** | 1. Parses data using `EventBridgeModel`.
2. Parses `detail` key using your model and returns it. | `BaseModel` +**SqsEnvelope** | 1. Parses data using `SqsModel`.
2. Parses records in `body` key using your model and return them in a list. | `List[BaseModel]` + +### Bringing your own envelope model + +You can create your own Envelope model and logic by inheriting from `BaseEnvelope`, and implementing the `parse` method. + +Here's an snippet of how the EventBridge Envelope we demonstrated previously is implemented. + +**EventBridge Model** + +```python:title=eventbridge_model.py +from datetime import datetime +from typing import Any, Dict, List + +from aws_lambda_powertools.utilities.parser import BaseModel, Field + + +class EventBridgeModel(BaseModel): + version: str + id: str # noqa: A003,VNE003 + source: str + account: str + time: datetime + region: str + resources: List[str] + detail_type: str = Field(None, alias="detail-type") + detail: Dict[str, Any] +``` + +**EventBridge Envelope** + +```python:title=eventbridge_envelope.py +from aws_lambda_powertools.utilities.parser import BaseEnvelope, BaseModel +from typing import Any, Dict +from ..models import EventBridgeModel + +class EventBridgeEnvelope(BaseEnvelope): # highlight-line + + def parse(self, data: Dict[str, Any], model: BaseModel) -> BaseModel: # highlight-line + """Parses data found with model provided + + Parameters + ---------- + data : Dict + Lambda event to be parsed + model : BaseModel + Data model provided to parse after extracting data using envelope + + Returns + ------- + Any + Parsed detail payload with model provided + """ + parsed_envelope = EventBridgeModel(**data) # highlight-line + return self._parse(data=parsed_envelope.detail, model=model) # highlight-line +``` + +**What's going on here, you might ask**: + +1. We defined an envelope named `EventBridgeEnvelope` inheriting from `BaseEnvelope` +2. Implemented the `parse` abstract method taking `data` and `model` as parameters +3. Then, we parsed the incoming data with our envelope model (EventBridgeModel) +3.1. This confirms that data received conforms with how EventBridge wraps events +4. Lastly, we call `_parse` from `BaseEnvelope` to parse the data in our envelope (.detail) using the customer model diff --git a/tests/functional/parser/test_dynamodb.py b/tests/functional/parser/test_dynamodb.py index ac5ebab40c3..d0e1cb87cf3 100644 --- a/tests/functional/parser/test_dynamodb.py +++ b/tests/functional/parser/test_dynamodb.py @@ -8,7 +8,7 @@ from tests.functional.parser.utils import load_event -@event_parser(model=MyDynamoBusiness, envelope=envelopes.DynamoDBEnvelope) +@event_parser(model=MyDynamoBusiness, envelope=envelopes.DynamoDBStreamEnvelope) def handle_dynamodb(event: List[Dict[str, MyDynamoBusiness]], _: LambdaContext): assert len(event) == 2 assert event[0]["OldImage"] is None From e1eac2d537d26547bda1df13d0a6e7da318660ce Mon Sep 17 00:00:00 2001 From: Heitor Lessa Date: Fri, 23 Oct 2020 09:29:32 +0200 Subject: [PATCH 10/26] chore: typo in list Co-authored-by: Joris Conijn --- docs/content/utilities/parser.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 8d8cbf11478..174b52733f5 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -273,7 +273,7 @@ assert ret.password1 == ret.password2 1. We imported built-in `models` from the parser utility 2. Used `envelopes.EventBridgeModel` as the envelope for our `UserModel` model 3. Parser parsed the original event against the EventBridge model -4 Parser then parsed the `detail` key using `UserModel` +4. Parser then parsed the `detail` key using `UserModel` ### Built-in envelopes From 4b6ecbf762b33318b50cee1aade14df9bc4da567 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Fri, 23 Oct 2020 10:40:56 +0200 Subject: [PATCH 11/26] docs: add extending built-in models --- docs/content/utilities/parser.mdx | 78 +++++++++++++++++++++++++++++-- 1 file changed, 74 insertions(+), 4 deletions(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 8d8cbf11478..834ab335592 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -59,7 +59,77 @@ These are simply Python classes that inherit from BaseModel. **Parser** enforces ### Extending built-in models -**TBW** +Parser comes with the following built-in models: + +Model name | Description +------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- +**DynamoDBStreamModel** | Lambda Event Source payload for Amazon DynamoDB Streams +**EventBridgeModel** | Lambda Event Source payload for Amazon EventBridge +**SqsModel** | Lambda Event Source payload for Amazon SQS + +You can extend them to include your own models, and yet have all other known fields parsed along the way. + +**EventBridge example** + +```python:title=extending_builtin_models.py +from aws_lambda_powertools.utilities.parser import parse, BaseModel +from aws_lambda_powertools.utilities.parser.models import EventBridgeModel + +from typing import List, Optional + +class OrderItem(BaseModel): + id: int + quantity: int + description: str + +class Order(BaseModel): + id: int + description: str + items: List[OrderItem] + +# highlight-start +class OrderEventModel(EventBridgeModel): + detail: Order +# highlight-end + +payload = { + "version": "0", + "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", + "detail-type": "OrderPurchased", + "source": "OrderService", + "account": "111122223333", + "time": "2020-10-22T18:43:48Z", + "region": "us-west-1", + "resources": ["some_additional"], + "detail": { # highlight-line + "id": 10876546789, + "description": "My order", + "items": [ + { + "id": 1015938732, + "quantity": 1, + "description": "item xpto" + } + ] + } +} + +ret = parse(model=OrderEventModel, event=payload) # highlight-line + +assert ret.source == "OrderService" +assert ret.detail.description == "My order" +assert ret.detail_type == "OrderPurchased" # we rename it to snake_case + +for order_item in ret.detail.items: + ... +``` + +**What's going on here, you might ask**: + +1. We imported our built-in model `EventBridgeModel` from the parser utility +2. Defined how our `Order` should look like +3. Defined how part of our EventBridge event should look like by overriding `detail` key within our `OrderEventModel` +4. Parser parsed the original event against `OrderEventModel` ## Parsing events @@ -232,7 +302,7 @@ parse(model=UserModel, event=payload) ## Envelopes -Envelope parameter is useful when your actual payload is wrapped around a known structure, for example Lambda Event Sources like Amazon EventBridge. +Envelope parameter is useful when your actual payload is wrapped around a known structure, for example Lambda Event Sources like EventBridge. Example of parsing a model found in an event coming from EventBridge, where all you want is what's inside the `detail` key. @@ -262,7 +332,7 @@ payload = { # highlight-end } -ret = parse(model=UserModel, envelope=envelopes.EventBridgeModel, payload=payload) # highlight-line +ret = parse(model=UserModel, envelope=envelopes.EventBridgeModel, event=payload) # highlight-line # Parsed model only contains our actual model, not the entire EventBridge + Payload parsed assert ret.password1 == ret.password2 @@ -270,7 +340,7 @@ assert ret.password1 == ret.password2 **What's going on here, you might ask**: -1. We imported built-in `models` from the parser utility +1. We imported built-in `envelopes` from the parser utility 2. Used `envelopes.EventBridgeModel` as the envelope for our `UserModel` model 3. Parser parsed the original event against the EventBridge model 4 Parser then parsed the `detail` key using `UserModel` From d434f48bd25e372437c49d4d0b3cd434b892b628 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Fri, 23 Oct 2020 10:56:22 +0200 Subject: [PATCH 12/26] docs: ensure examples can be copied/pasted as-is Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 36 +++++++++++++++++++++++++------ 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 834ab335592..49cc0554351 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -118,7 +118,7 @@ ret = parse(model=OrderEventModel, event=payload) # highlight-line assert ret.source == "OrderService" assert ret.detail.description == "My order" -assert ret.detail_type == "OrderPurchased" # we rename it to snake_case +assert ret.detail_type == "OrderPurchased" # we rename it to snake_case since detail-type is an invalid name for order_item in ret.detail.items: ... @@ -142,11 +142,21 @@ Use the decorator for fail fast scenarios where you want your Lambda function to `event_parser` decorator will throw a `ModelValidationError` if your event cannot be parsed according to the model. ```python=:title=event_parser_decorator.py -from aws_lambda_powertools.utilities.parser import parse, ModelValidationError +from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, ModelValidationError from aws_lambda_powertools.utilities.typing import LambdaContext import json -# Raw event for the Order model we've defined earlier +class OrderItem(BaseModel): + id: int + quantity: int + description: str + +class Order(BaseModel): + id: int + description: str + items: List[OrderItem] # nesting models are supported + optional_field: Optional[str] # this field may or may not be available when parsing + payload = { "id": 10876546789, "description": "My order", @@ -177,15 +187,27 @@ handler(event=json.dumps(payload), context=LambdaContext()) # also works if even Use this standalone function when you want more control over the data validation process, for example returning a 400 error for malformed payloads. ```python:title=parse_standalone_example.py -from aws_lambda_powertools.utilities.parser import parse, ModelValidationError +from aws_lambda_powertools.utilities.parser import parse, BaseModel, ModelValidationError + +class OrderItem(BaseModel): + id: int + quantity: int + description: str + +class Order(BaseModel): + id: int + description: str + items: List[OrderItem] # nesting models are supported + optional_field: Optional[str] # this field may or may not be available when parsing + -# Raw event for the Order model we've defined earlier payload = { "id": 10876546789, "description": "My order", "items": [ { - "id": 1015938732, + # this will cause a validation error + "id": [1015938732], # highlight-line "quantity": 1, "description": "item xpto" } @@ -194,7 +216,7 @@ payload = { def my_function(): try: - parsed_payload: Order = parse(event=payload, model=HelloWorldModel) # highlight-line + parsed_payload: Order = parse(event=payload, model=Order) # highlight-line # payload dict is now parsed into our model return parsed_payload.items except ModelValidationError: From 4e738e9065bce6f5cb6d07e70b89193a1bb0b4b4 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Fri, 23 Oct 2020 12:15:01 +0200 Subject: [PATCH 13/26] improv: export Pydantic ValidationError instead of our own --- .../utilities/parser/__init__.py | 5 ++--- .../utilities/parser/exceptions.py | 4 ---- .../utilities/parser/parser.py | 18 +++++++----------- docs/content/utilities/parser.mdx | 8 ++++---- tests/functional/parser/conftest.py | 9 +++------ tests/functional/parser/test_dynamodb.py | 6 +++--- tests/functional/parser/test_eventbridge.py | 6 +++--- tests/functional/parser/test_parser.py | 4 ++-- tests/functional/parser/test_sqs.py | 6 +++--- 9 files changed, 27 insertions(+), 39 deletions(-) diff --git a/aws_lambda_powertools/utilities/parser/__init__.py b/aws_lambda_powertools/utilities/parser/__init__.py index a010301e7fb..1bc67934b13 100644 --- a/aws_lambda_powertools/utilities/parser/__init__.py +++ b/aws_lambda_powertools/utilities/parser/__init__.py @@ -2,9 +2,8 @@ """ from . import envelopes from .envelopes import BaseEnvelope -from .exceptions import ModelValidationError from .parser import event_parser, parse -from .pydantic import BaseModel, Field, root_validator, validator +from .pydantic import BaseModel, Field, ValidationError, root_validator, validator __all__ = [ "event_parser", @@ -15,5 +14,5 @@ "Field", "validator", "root_validator", - "ModelValidationError", + "ValidationError", ] diff --git a/aws_lambda_powertools/utilities/parser/exceptions.py b/aws_lambda_powertools/utilities/parser/exceptions.py index 93e259df371..0df217e8522 100644 --- a/aws_lambda_powertools/utilities/parser/exceptions.py +++ b/aws_lambda_powertools/utilities/parser/exceptions.py @@ -2,9 +2,5 @@ class InvalidEnvelopeError(Exception): """Input envelope is not callable and instance of BaseEnvelope""" -class ModelValidationError(Exception): - """Input data does not conform with model""" - - class InvalidModelTypeError(Exception): """Input data model does not implement BaseModel""" diff --git a/aws_lambda_powertools/utilities/parser/parser.py b/aws_lambda_powertools/utilities/parser/parser.py index 1f3a2d4ace0..aed79ebec6a 100644 --- a/aws_lambda_powertools/utilities/parser/parser.py +++ b/aws_lambda_powertools/utilities/parser/parser.py @@ -1,12 +1,12 @@ import logging from typing import Any, Callable, Dict, Optional -from pydantic import BaseModel, ValidationError +from pydantic import BaseModel from ...middleware_factory import lambda_handler_decorator from ..typing import LambdaContext from .envelopes.base import BaseEnvelope -from .exceptions import InvalidEnvelopeError, InvalidModelTypeError, ModelValidationError +from .exceptions import InvalidEnvelopeError, InvalidModelTypeError logger = logging.getLogger(__name__) @@ -72,7 +72,7 @@ def handler(event: Order, context: LambdaContext): Raises ------ - ModelValidationError + ValidationError When input event does not conform with model provided InvalidModelTypeError When model given does not implement BaseModel @@ -94,7 +94,7 @@ def parse(event: Dict[str, Any], model: BaseModel, envelope: Optional[BaseEnvelo **Lambda handler decorator to parse & validate event** - from aws_lambda_powertools.utilities.parser.exceptions import ModelValidationError + from aws_lambda_powertools.utilities.parser import ValidationError class Order(BaseModel): id: int @@ -104,7 +104,7 @@ class Order(BaseModel): def handler(event: Order, context: LambdaContext): try: parse(model=Order) - except ModelValidationError: + except ValidationError: ... **Lambda handler decorator to parse & validate event - using built-in envelope** @@ -117,7 +117,7 @@ class Order(BaseModel): def handler(event: Order, context: LambdaContext): try: parse(model=Order, envelope=envelopes.EVENTBRIDGE) - except ModelValidationError: + except ValidationError: ... Parameters @@ -131,7 +131,7 @@ def handler(event: Order, context: LambdaContext): Raises ------ - ModelValidationError + ValidationError When input event does not conform with model provided InvalidModelTypeError When model given does not implement BaseModel @@ -144,8 +144,6 @@ def handler(event: Order, context: LambdaContext): return envelope().parse(data=event, model=model) except AttributeError: raise InvalidEnvelopeError(f"Envelope must implement BaseEnvelope, envelope={envelope}") - except (ValidationError, TypeError) as e: - raise ModelValidationError(f"Input event does not conform with model, envelope={envelope}") from e try: logger.debug("Parsing and validating event model; no envelope used") @@ -153,7 +151,5 @@ def handler(event: Order, context: LambdaContext): return model.parse_raw(event) return model.parse_obj(event) - except (ValidationError, TypeError) as e: - raise ModelValidationError("Input event does not conform with model") from e except AttributeError: raise InvalidModelTypeError("Input model must implement BaseModel") diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 3d86ccbbb6d..468afd0099f 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -139,10 +139,10 @@ You can parse inbound events using **event_parser** decorator, or the standalone Use the decorator for fail fast scenarios where you want your Lambda function to raise an exception in the event of a malformed payload. -`event_parser` decorator will throw a `ModelValidationError` if your event cannot be parsed according to the model. +`event_parser` decorator will throw a `ValidationError` if your event cannot be parsed according to the model. ```python=:title=event_parser_decorator.py -from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, ModelValidationError +from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, ValidationError from aws_lambda_powertools.utilities.typing import LambdaContext import json @@ -187,7 +187,7 @@ handler(event=json.dumps(payload), context=LambdaContext()) # also works if even Use this standalone function when you want more control over the data validation process, for example returning a 400 error for malformed payloads. ```python:title=parse_standalone_example.py -from aws_lambda_powertools.utilities.parser import parse, BaseModel, ModelValidationError +from aws_lambda_powertools.utilities.parser import parse, BaseModel, ValidationError class OrderItem(BaseModel): id: int @@ -219,7 +219,7 @@ def my_function(): parsed_payload: Order = parse(event=payload, model=Order) # highlight-line # payload dict is now parsed into our model return parsed_payload.items - except ModelValidationError: + except ValidationError: return { "status_code": 400, "message": "Invalid order" diff --git a/tests/functional/parser/conftest.py b/tests/functional/parser/conftest.py index 27fd4b2d1f6..34199a322b2 100644 --- a/tests/functional/parser/conftest.py +++ b/tests/functional/parser/conftest.py @@ -1,9 +1,9 @@ from typing import Any, Dict import pytest -from pydantic import BaseModel, ValidationError +from pydantic import BaseModel -from aws_lambda_powertools.utilities.parser import BaseEnvelope, ModelValidationError +from aws_lambda_powertools.utilities.parser import BaseEnvelope @pytest.fixture @@ -37,10 +37,7 @@ class MyDummyEnvelope(BaseEnvelope): """Unwrap dummy event within payload key""" def parse(self, data: Dict[str, Any], model: BaseModel): - try: - parsed_enveloped = dummy_envelope_schema(**data) - except (ValidationError, TypeError) as e: - raise ModelValidationError("Dummy input does not conform with schema") from e + parsed_enveloped = dummy_envelope_schema(**data) return self._parse(data=parsed_enveloped.payload, model=model) return MyDummyEnvelope diff --git a/tests/functional/parser/test_dynamodb.py b/tests/functional/parser/test_dynamodb.py index d0e1cb87cf3..bd7e0795f42 100644 --- a/tests/functional/parser/test_dynamodb.py +++ b/tests/functional/parser/test_dynamodb.py @@ -2,7 +2,7 @@ import pytest -from aws_lambda_powertools.utilities.parser import envelopes, event_parser, exceptions +from aws_lambda_powertools.utilities.parser import ValidationError, envelopes, event_parser from aws_lambda_powertools.utilities.typing import LambdaContext from tests.functional.parser.schemas import MyAdvancedDynamoBusiness, MyDynamoBusiness from tests.functional.parser.utils import load_event @@ -59,11 +59,11 @@ def test_dynamo_db_stream_trigger_event_no_envelope(): def test_validate_event_does_not_conform_with_model_no_envelope(): event_dict: Any = {"hello": "s"} - with pytest.raises(exceptions.ModelValidationError): + with pytest.raises(ValidationError): handle_dynamodb_no_envelope(event_dict, LambdaContext()) def test_validate_event_does_not_conform_with_model(): event_dict: Any = {"hello": "s"} - with pytest.raises(exceptions.ModelValidationError): + with pytest.raises(ValidationError): handle_dynamodb(event_dict, LambdaContext()) diff --git a/tests/functional/parser/test_eventbridge.py b/tests/functional/parser/test_eventbridge.py index 07387e9ba0a..1af481bc52d 100644 --- a/tests/functional/parser/test_eventbridge.py +++ b/tests/functional/parser/test_eventbridge.py @@ -2,7 +2,7 @@ import pytest -from aws_lambda_powertools.utilities.parser import envelopes, event_parser, exceptions +from aws_lambda_powertools.utilities.parser import ValidationError, envelopes, event_parser from aws_lambda_powertools.utilities.typing import LambdaContext from tests.functional.parser.schemas import MyAdvancedEventbridgeBusiness, MyEventbridgeBusiness from tests.functional.parser.utils import load_event @@ -46,7 +46,7 @@ def test_validate_event_does_not_conform_with_user_dict_model(): "resources": ["arn:aws:ec2:us-west-1:123456789012:instance/i-1234567890abcdef0"], "detail": {}, } - with pytest.raises(exceptions.ModelValidationError) as e: + with pytest.raises(ValidationError) as e: handle_eventbridge(event_dict, LambdaContext()) print(e.exconly()) @@ -57,5 +57,5 @@ def test_handle_eventbridge_trigger_event_no_envelope(): def test_handle_invalid_event_with_eventbridge_envelope(): - with pytest.raises(exceptions.ModelValidationError): + with pytest.raises(ValidationError): handle_eventbridge(event={}, context=LambdaContext()) diff --git a/tests/functional/parser/test_parser.py b/tests/functional/parser/test_parser.py index ec802471585..5e9e40faec4 100644 --- a/tests/functional/parser/test_parser.py +++ b/tests/functional/parser/test_parser.py @@ -3,7 +3,7 @@ import pytest -from aws_lambda_powertools.utilities.parser import event_parser, exceptions +from aws_lambda_powertools.utilities.parser import ValidationError, event_parser, exceptions from aws_lambda_powertools.utilities.typing import LambdaContext @@ -13,7 +13,7 @@ def test_parser_unsupported_event(dummy_schema, invalid_value): def handle_no_envelope(event: Dict, _: LambdaContext): return event - with pytest.raises(exceptions.ModelValidationError): + with pytest.raises(ValidationError): handle_no_envelope(event=invalid_value, context=LambdaContext()) diff --git a/tests/functional/parser/test_sqs.py b/tests/functional/parser/test_sqs.py index 2ee992e2fa1..0cea8246b50 100644 --- a/tests/functional/parser/test_sqs.py +++ b/tests/functional/parser/test_sqs.py @@ -2,7 +2,7 @@ import pytest -from aws_lambda_powertools.utilities.parser import envelopes, event_parser, exceptions +from aws_lambda_powertools.utilities.parser import ValidationError, envelopes, event_parser from aws_lambda_powertools.utilities.typing import LambdaContext from tests.functional.parser.schemas import MyAdvancedSqsBusiness, MySqsBusiness from tests.functional.parser.utils import load_event @@ -23,7 +23,7 @@ def test_handle_sqs_trigger_event_json_body(sqs_event): # noqa: F811 def test_validate_event_does_not_conform_with_model(): event: Any = {"invalid": "event"} - with pytest.raises(exceptions.ModelValidationError): + with pytest.raises(ValidationError): handle_sqs_json_body(event, LambdaContext()) @@ -51,7 +51,7 @@ def test_validate_event_does_not_conform_user_json_string_with_model(): ] } - with pytest.raises(exceptions.ModelValidationError): + with pytest.raises(ValidationError): handle_sqs_json_body(event, LambdaContext()) From 6c359d569bc3cba7c09e97b78cf89d79a4346459 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Fri, 23 Oct 2020 13:30:22 +0200 Subject: [PATCH 14/26] fix: remove malformed 3.1. sentence Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 468afd0099f..f38ca7bec99 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -436,6 +436,5 @@ class EventBridgeEnvelope(BaseEnvelope): # highlight-line 1. We defined an envelope named `EventBridgeEnvelope` inheriting from `BaseEnvelope` 2. Implemented the `parse` abstract method taking `data` and `model` as parameters -3. Then, we parsed the incoming data with our envelope model (EventBridgeModel) -3.1. This confirms that data received conforms with how EventBridge wraps events +3. Then, we parsed the incoming data with our envelope to confirm it matches EventBridge's structure defined in `EventBridgeModel` 4. Lastly, we call `_parse` from `BaseEnvelope` to parse the data in our envelope (.detail) using the customer model From 9b9f4f115807bcbb9f32ec2485ffcba33aa0de20 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Fri, 23 Oct 2020 13:38:33 +0200 Subject: [PATCH 15/26] fix: debug logging in envelopes before each parsing --- aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py | 2 ++ .../utilities/parser/envelopes/event_bridge.py | 2 ++ aws_lambda_powertools/utilities/parser/envelopes/sqs.py | 2 ++ 3 files changed, 6 insertions(+) diff --git a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py index 69585e02e15..b586bc8353f 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py @@ -34,8 +34,10 @@ def parse(self, data: Dict[str, Any], model: BaseModel) -> List[Dict[Literal["Ne """ + logger.debug(f"Parsing incoming data with DynamoDB Stream model {DynamoDBStreamModel}") parsed_envelope = DynamoDBStreamModel(**data) output = [] + logger.debug(f"Parsing DynamoDB Stream new and old records with {model}") for record in parsed_envelope.Records: output.append( { diff --git a/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py b/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py index 8b91266e848..6757a724f68 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py @@ -27,5 +27,7 @@ def parse(self, data: Dict[str, Any], model: BaseModel) -> BaseModel: Any Parsed detail payload with model provided """ + logger.debug(f"Parsing incoming data with EventBridge model {EventBridgeModel}") parsed_envelope = EventBridgeModel(**data) + logger.debug(f"Parsing event payload in `detail` with {model}") return self._parse(data=parsed_envelope.detail, model=model) diff --git a/aws_lambda_powertools/utilities/parser/envelopes/sqs.py b/aws_lambda_powertools/utilities/parser/envelopes/sqs.py index 61d9a4b40c2..4906897265d 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/sqs.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/sqs.py @@ -34,8 +34,10 @@ def parse(self, data: Dict[str, Any], model: Union[BaseModel, str]) -> List[Base List List of records parsed with model provided """ + logger.debug(f"Parsing incoming data with SQS model {SqsModel}") parsed_envelope = SqsModel(**data) output = [] + logger.debug(f"Parsing SQS records in `body` with {model}") for record in parsed_envelope.Records: output.append(self._parse(record.body, model)) return output From 27327604d697cf080547c5988920f6cb429bad0e Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Fri, 23 Oct 2020 14:03:45 +0200 Subject: [PATCH 16/26] chore: spacing --- aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py index b586bc8353f..61c35d81c58 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py @@ -31,8 +31,6 @@ def parse(self, data: Dict[str, Any], model: BaseModel) -> List[Dict[Literal["Ne ------- List List of records parsed with model provided - - """ logger.debug(f"Parsing incoming data with DynamoDB Stream model {DynamoDBStreamModel}") parsed_envelope = DynamoDBStreamModel(**data) From 5f1ad0af7b09489900a6bea5cc831b4950a8b85a Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Fri, 23 Oct 2020 14:04:54 +0200 Subject: [PATCH 17/26] fix: generic type to match ABC bound class --- aws_lambda_powertools/utilities/parser/parser.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/aws_lambda_powertools/utilities/parser/parser.py b/aws_lambda_powertools/utilities/parser/parser.py index aed79ebec6a..c3394a8bd5d 100644 --- a/aws_lambda_powertools/utilities/parser/parser.py +++ b/aws_lambda_powertools/utilities/parser/parser.py @@ -1,5 +1,5 @@ import logging -from typing import Any, Callable, Dict, Optional +from typing import Any, Callable, Dict, Optional, Type, TypeVar from pydantic import BaseModel @@ -8,6 +8,8 @@ from .envelopes.base import BaseEnvelope from .exceptions import InvalidEnvelopeError, InvalidModelTypeError +Model = TypeVar("Model", bound=BaseModel) +Envelope = TypeVar("Envelope", bound=BaseEnvelope) logger = logging.getLogger(__name__) @@ -16,8 +18,8 @@ def event_parser( handler: Callable[[Dict, Any], Any], event: Dict[str, Any], context: LambdaContext, - model: BaseModel, - envelope: Optional[BaseEnvelope] = None, + model: Type[Model], + envelope: Optional[Type[Envelope]] = None, ) -> Any: """Lambda handler decorator to parse & validate events using Pydantic models @@ -84,7 +86,7 @@ def handler(event: Order, context: LambdaContext): return handler(parsed_event, context) -def parse(event: Dict[str, Any], model: BaseModel, envelope: Optional[BaseEnvelope] = None) -> Any: +def parse(event: Dict[str, Any], model: Type[Model], envelope: Optional[Type[Envelope]] = None) -> Any: """Standalone function to parse & validate events using Pydantic models Typically used when you need fine-grained control over error handling compared to event_parser decorator. @@ -152,4 +154,4 @@ def handler(event: Order, context: LambdaContext): return model.parse_obj(event) except AttributeError: - raise InvalidModelTypeError("Input model must implement BaseModel") + raise InvalidModelTypeError(f"Input model must implement BaseModel, model={model}") From d2148f2264439c5338da2b67545bfd46d8048c9e Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Fri, 23 Oct 2020 14:14:37 +0200 Subject: [PATCH 18/26] docs: add a FAQ section Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index f38ca7bec99..d43b49fe0e2 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -438,3 +438,21 @@ class EventBridgeEnvelope(BaseEnvelope): # highlight-line 2. Implemented the `parse` abstract method taking `data` and `model` as parameters 3. Then, we parsed the incoming data with our envelope to confirm it matches EventBridge's structure defined in `EventBridgeModel` 4. Lastly, we call `_parse` from `BaseEnvelope` to parse the data in our envelope (.detail) using the customer model + +## FAQ + +**When should I use parser vs data_classes utility?** + +Use data classes utility when you're after autocomplete, self-documented attributes and helpers to extract data from common event sources. + +Parser is best suited for those looking for a trade-off between defining their models for deep validation, parsing and autocomplete for an additional dependency to be brought in. + +**How do I import X from Pydantic?** + +We export most common classes, exceptions, and utilities from Pydantic as part of parser e.g. `from aws_lambda_powertools.utilities.parser import BaseModel`. + +If what's your trying to use isn't available as part of the high level import system, use the following escape hatch mechanism: + +```python:title=escape_hatch.py +from aws_lambda_powertools.utilities.parser.pydantic import +``` From 234afd92c59057614d5e9ed68dedbfb81dd434fe Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Fri, 23 Oct 2020 14:49:00 +0200 Subject: [PATCH 19/26] docs: add cold start data Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 50 +++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index d43b49fe0e2..308a991e8e8 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -456,3 +456,53 @@ If what's your trying to use isn't available as part of the high level import sy ```python:title=escape_hatch.py from aws_lambda_powertools.utilities.parser.pydantic import ``` + +**What is the cold start impact in bringing this additional dependency?** + +No significant cold start impact. It does increase the final uncompressed package by **71M**, when you bring the additional dependency that parser requires. + +Artillery load test sample against a [hello world sample](https://github.com/aws-samples/cookiecutter-aws-sam-python) using Tracer, Metrics, and Logger with and without parser. + +**No parser** + +> **Uncompressed package size**: 55M, **p99**: 180.3ms + +``` +Summary report @ 14:36:07(+0200) 2020-10-23 + Scenarios launched: 10 + Scenarios completed: 10 + Requests completed: 2000 + Mean response/sec: 114.81 + Response time (msec): + min: 54.9 + max: 1684.9 + median: 68 + p95: 109.1 + p99: 180.3 + Scenario counts: + 0: 10 (100%) + Codes: + 200: 2000 +``` + +**With parser** + +> **Uncompressed package size**: 128M, **p99**: 193.1ms + +``` +Summary report @ 14:29:23(+0200) 2020-10-23 + Scenarios launched: 10 + Scenarios completed: 10 + Requests completed: 2000 + Mean response/sec: 111.67 + Response time (msec): + min: 54.3 + max: 1887.2 + median: 66.1 + p95: 113.3 + p99: 193.1 + Scenario counts: + 0: 10 (100%) + Codes: + 200: 2000 +``` From 3750d5584d7e9b90c43c25387831bc1ada95fca6 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Sat, 24 Oct 2020 21:40:04 +0200 Subject: [PATCH 20/26] improv: address Koudai's PR feedback --- .../utilities/parser/envelopes/base.py | 15 ++++++++---- .../utilities/parser/envelopes/dynamodb.py | 12 +++++----- .../parser/envelopes/event_bridge.py | 9 ++++---- .../utilities/parser/envelopes/sqs.py | 13 +++++------ .../utilities/parser/parser.py | 23 ++++++++----------- .../utilities/parser/types.py | 6 +++++ 6 files changed, 42 insertions(+), 36 deletions(-) create mode 100644 aws_lambda_powertools/utilities/parser/types.py diff --git a/aws_lambda_powertools/utilities/parser/envelopes/base.py b/aws_lambda_powertools/utilities/parser/envelopes/base.py index baf6cd33420..f2a59247f11 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/base.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/base.py @@ -1,8 +1,8 @@ import logging from abc import ABC, abstractmethod -from typing import Any, Dict, Union +from typing import Any, Dict, TypeVar, Union -from pydantic import BaseModel +from ..types import Model logger = logging.getLogger(__name__) @@ -11,14 +11,14 @@ class BaseEnvelope(ABC): """ABC implementation for creating a supported Envelope""" @staticmethod - def _parse(data: Union[Dict[str, Any], str], model: BaseModel) -> Any: + def _parse(data: Union[Dict[str, Any], str], model: Model) -> Model: """Parses envelope data against model provided Parameters ---------- data : Dict Data to be parsed and validated - model + model : Model Data model to parse and validate data against Returns @@ -38,7 +38,7 @@ def _parse(data: Union[Dict[str, Any], str], model: BaseModel) -> Any: return model.parse_obj(data) @abstractmethod - def parse(self, data: Dict[str, Any], model: BaseModel): + def parse(self, data: Dict[str, Any], model: Model): """Implementation to parse data against envelope model, then against the data model NOTE: Call `_parse` method to fully parse data with model provided. @@ -56,3 +56,8 @@ def parse(...): return self._parse(data=parsed_envelope.detail, model=data_model) """ return NotImplemented # pragma: no cover + + +# Generic to support type annotations throughout parser +# Note: Can't be defined under types.py due to circular dependency +Envelope = TypeVar("Envelope", bound=BaseEnvelope) diff --git a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py index 61c35d81c58..10757f3bdf5 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py @@ -1,10 +1,10 @@ import logging from typing import Any, Dict, List -from pydantic import BaseModel from typing_extensions import Literal from ..models import DynamoDBStreamModel +from ..types import Model from .base import BaseEnvelope logger = logging.getLogger(__name__) @@ -17,14 +17,14 @@ class DynamoDBStreamEnvelope(BaseEnvelope): length of the list is the record's amount in the original event. """ - def parse(self, data: Dict[str, Any], model: BaseModel) -> List[Dict[Literal["NewImage", "OldImage"], BaseModel]]: + def parse(self, data: Dict[str, Any], model: Model) -> List[Dict[Literal["NewImage", "OldImage"], Model]]: """Parses DynamoDB Stream records found in either NewImage and OldImage with model provided Parameters ---------- data : Dict Lambda event to be parsed - model : BaseModel + model : Model Data model provided to parse after extracting data using envelope Returns @@ -33,14 +33,14 @@ def parse(self, data: Dict[str, Any], model: BaseModel) -> List[Dict[Literal["Ne List of records parsed with model provided """ logger.debug(f"Parsing incoming data with DynamoDB Stream model {DynamoDBStreamModel}") - parsed_envelope = DynamoDBStreamModel(**data) + parsed_envelope = DynamoDBStreamModel.parse_obj(data) output = [] logger.debug(f"Parsing DynamoDB Stream new and old records with {model}") for record in parsed_envelope.Records: output.append( { - "NewImage": self._parse(record.dynamodb.NewImage, model), - "OldImage": self._parse(record.dynamodb.OldImage, model), + "NewImage": self._parse(data=record.dynamodb.NewImage, model=model), + "OldImage": self._parse(data=record.dynamodb.OldImage, model=model), } ) # noinspection PyTypeChecker diff --git a/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py b/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py index 6757a724f68..04133a70786 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py @@ -1,9 +1,8 @@ import logging from typing import Any, Dict -from pydantic import BaseModel - from ..models import EventBridgeModel +from ..types import Model from .base import BaseEnvelope logger = logging.getLogger(__name__) @@ -12,14 +11,14 @@ class EventBridgeEnvelope(BaseEnvelope): """EventBridge envelope to extract data within detail key""" - def parse(self, data: Dict[str, Any], model: BaseModel) -> BaseModel: + def parse(self, data: Dict[str, Any], model: Model) -> Model: """Parses data found with model provided Parameters ---------- data : Dict Lambda event to be parsed - model : BaseModel + model : Model Data model provided to parse after extracting data using envelope Returns @@ -28,6 +27,6 @@ def parse(self, data: Dict[str, Any], model: BaseModel) -> BaseModel: Parsed detail payload with model provided """ logger.debug(f"Parsing incoming data with EventBridge model {EventBridgeModel}") - parsed_envelope = EventBridgeModel(**data) + parsed_envelope = EventBridgeModel.parse_obj(data) logger.debug(f"Parsing event payload in `detail` with {model}") return self._parse(data=parsed_envelope.detail, model=model) diff --git a/aws_lambda_powertools/utilities/parser/envelopes/sqs.py b/aws_lambda_powertools/utilities/parser/envelopes/sqs.py index 4906897265d..e2cd4542265 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/sqs.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/sqs.py @@ -1,9 +1,8 @@ import logging -from typing import Any, Dict, List, Union - -from pydantic import BaseModel +from typing import Any, Dict, List from ..models import SqsModel +from ..types import Model from .base import BaseEnvelope logger = logging.getLogger(__name__) @@ -19,14 +18,14 @@ class SqsEnvelope(BaseEnvelope): all items in the list will be parsed as str and npt as JSON (and vice versa) """ - def parse(self, data: Dict[str, Any], model: Union[BaseModel, str]) -> List[BaseModel]: + def parse(self, data: Dict[str, Any], model: Model) -> List[Model]: """Parses records found with model provided Parameters ---------- data : Dict Lambda event to be parsed - model : BaseModel + model : Model Data model provided to parse after extracting data using envelope Returns @@ -35,9 +34,9 @@ def parse(self, data: Dict[str, Any], model: Union[BaseModel, str]) -> List[Base List of records parsed with model provided """ logger.debug(f"Parsing incoming data with SQS model {SqsModel}") - parsed_envelope = SqsModel(**data) + parsed_envelope = SqsModel.parse_obj(data) output = [] logger.debug(f"Parsing SQS records in `body` with {model}") for record in parsed_envelope.Records: - output.append(self._parse(record.body, model)) + output.append(self._parse(data=record.body, model=model)) return output diff --git a/aws_lambda_powertools/utilities/parser/parser.py b/aws_lambda_powertools/utilities/parser/parser.py index c3394a8bd5d..1983b54dfbb 100644 --- a/aws_lambda_powertools/utilities/parser/parser.py +++ b/aws_lambda_powertools/utilities/parser/parser.py @@ -1,15 +1,12 @@ import logging -from typing import Any, Callable, Dict, Optional, Type, TypeVar - -from pydantic import BaseModel +from typing import Any, Callable, Dict, Optional from ...middleware_factory import lambda_handler_decorator from ..typing import LambdaContext -from .envelopes.base import BaseEnvelope +from .envelopes.base import Envelope from .exceptions import InvalidEnvelopeError, InvalidModelTypeError +from .types import Model -Model = TypeVar("Model", bound=BaseModel) -Envelope = TypeVar("Envelope", bound=BaseEnvelope) logger = logging.getLogger(__name__) @@ -18,8 +15,8 @@ def event_parser( handler: Callable[[Dict, Any], Any], event: Dict[str, Any], context: LambdaContext, - model: Type[Model], - envelope: Optional[Type[Envelope]] = None, + model: Model, + envelope: Optional[Envelope] = None, ) -> Any: """Lambda handler decorator to parse & validate events using Pydantic models @@ -67,9 +64,9 @@ def handler(event: Order, context: LambdaContext): Lambda event to be parsed & validated context: LambdaContext Lambda context object - model: BaseModel + model: Model Your data model that will replace the event. - envelope: BaseEnvelope + envelope: Envelope Optional envelope to extract the model from Raises @@ -86,7 +83,7 @@ def handler(event: Order, context: LambdaContext): return handler(parsed_event, context) -def parse(event: Dict[str, Any], model: Type[Model], envelope: Optional[Type[Envelope]] = None) -> Any: +def parse(event: Dict[str, Any], model: Model, envelope: Optional[Envelope] = None) -> Model: """Standalone function to parse & validate events using Pydantic models Typically used when you need fine-grained control over error handling compared to event_parser decorator. @@ -126,9 +123,9 @@ def handler(event: Order, context: LambdaContext): ---------- event: Dict Lambda event to be parsed & validated - model: BaseModel + model: Model Your data model that will replace the event - envelope: BaseEnvelope + envelope: Envelope Optional envelope to extract the model from Raises diff --git a/aws_lambda_powertools/utilities/parser/types.py b/aws_lambda_powertools/utilities/parser/types.py new file mode 100644 index 00000000000..cada13b18e3 --- /dev/null +++ b/aws_lambda_powertools/utilities/parser/types.py @@ -0,0 +1,6 @@ +"""Generics and other shared types used across parser""" +from typing import TypeVar + +from pydantic import BaseModel + +Model = TypeVar("Model", bound=BaseModel) From ffbde1d7c72161242d80b168d6a9b4bcc86b059c Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Sat, 24 Oct 2020 21:42:41 +0200 Subject: [PATCH 21/26] fix: _parse return type --- aws_lambda_powertools/utilities/parser/envelopes/base.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/aws_lambda_powertools/utilities/parser/envelopes/base.py b/aws_lambda_powertools/utilities/parser/envelopes/base.py index f2a59247f11..b3b08a22c0c 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/base.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/base.py @@ -11,7 +11,7 @@ class BaseEnvelope(ABC): """ABC implementation for creating a supported Envelope""" @staticmethod - def _parse(data: Union[Dict[str, Any], str], model: Model) -> Model: + def _parse(data: Union[Dict[str, Any], str], model: Model) -> Union[Model, None]: """Parses envelope data against model provided Parameters From 33fec718a0bc3d0b749b14238436c909450c421f Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Sun, 25 Oct 2020 14:42:12 +0100 Subject: [PATCH 22/26] improv: address Koudai's PR feedback on mypy --- aws_lambda_powertools/utilities/parser/envelopes/base.py | 6 +++--- .../utilities/parser/envelopes/dynamodb.py | 9 +++------ .../utilities/parser/envelopes/event_bridge.py | 4 ++-- aws_lambda_powertools/utilities/parser/envelopes/sqs.py | 4 ++-- aws_lambda_powertools/utilities/parser/parser.py | 2 +- 5 files changed, 11 insertions(+), 14 deletions(-) diff --git a/aws_lambda_powertools/utilities/parser/envelopes/base.py b/aws_lambda_powertools/utilities/parser/envelopes/base.py index b3b08a22c0c..4cf9c1b94ec 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/base.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/base.py @@ -1,6 +1,6 @@ import logging from abc import ABC, abstractmethod -from typing import Any, Dict, TypeVar, Union +from typing import Any, Dict, Optional, TypeVar, Union from ..types import Model @@ -11,7 +11,7 @@ class BaseEnvelope(ABC): """ABC implementation for creating a supported Envelope""" @staticmethod - def _parse(data: Union[Dict[str, Any], str], model: Model) -> Union[Model, None]: + def _parse(data: Optional[Union[Dict[str, Any], Any]], model: Model) -> Union[Model, None]: """Parses envelope data against model provided Parameters @@ -38,7 +38,7 @@ def _parse(data: Union[Dict[str, Any], str], model: Model) -> Union[Model, None] return model.parse_obj(data) @abstractmethod - def parse(self, data: Dict[str, Any], model: Model): + def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: Model): """Implementation to parse data against envelope model, then against the data model NOTE: Call `_parse` method to fully parse data with model provided. diff --git a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py index 10757f3bdf5..38d19ffe1c6 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/dynamodb.py @@ -1,7 +1,5 @@ import logging -from typing import Any, Dict, List - -from typing_extensions import Literal +from typing import Any, Dict, List, Optional, Union from ..models import DynamoDBStreamModel from ..types import Model @@ -17,7 +15,7 @@ class DynamoDBStreamEnvelope(BaseEnvelope): length of the list is the record's amount in the original event. """ - def parse(self, data: Dict[str, Any], model: Model) -> List[Dict[Literal["NewImage", "OldImage"], Model]]: + def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: Model) -> List[Dict[str, Optional[Model]]]: """Parses DynamoDB Stream records found in either NewImage and OldImage with model provided Parameters @@ -30,7 +28,7 @@ def parse(self, data: Dict[str, Any], model: Model) -> List[Dict[Literal["NewIma Returns ------- List - List of records parsed with model provided + List of dictionaries with NewImage and OldImage records parsed with model provided """ logger.debug(f"Parsing incoming data with DynamoDB Stream model {DynamoDBStreamModel}") parsed_envelope = DynamoDBStreamModel.parse_obj(data) @@ -43,5 +41,4 @@ def parse(self, data: Dict[str, Any], model: Model) -> List[Dict[Literal["NewIma "OldImage": self._parse(data=record.dynamodb.OldImage, model=model), } ) - # noinspection PyTypeChecker return output diff --git a/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py b/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py index 04133a70786..5b40926b482 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/event_bridge.py @@ -1,5 +1,5 @@ import logging -from typing import Any, Dict +from typing import Any, Dict, Optional, Union from ..models import EventBridgeModel from ..types import Model @@ -11,7 +11,7 @@ class EventBridgeEnvelope(BaseEnvelope): """EventBridge envelope to extract data within detail key""" - def parse(self, data: Dict[str, Any], model: Model) -> Model: + def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: Model) -> Optional[Model]: """Parses data found with model provided Parameters diff --git a/aws_lambda_powertools/utilities/parser/envelopes/sqs.py b/aws_lambda_powertools/utilities/parser/envelopes/sqs.py index e2cd4542265..3ed479ffa31 100644 --- a/aws_lambda_powertools/utilities/parser/envelopes/sqs.py +++ b/aws_lambda_powertools/utilities/parser/envelopes/sqs.py @@ -1,5 +1,5 @@ import logging -from typing import Any, Dict, List +from typing import Any, Dict, List, Optional, Union from ..models import SqsModel from ..types import Model @@ -18,7 +18,7 @@ class SqsEnvelope(BaseEnvelope): all items in the list will be parsed as str and npt as JSON (and vice versa) """ - def parse(self, data: Dict[str, Any], model: Model) -> List[Model]: + def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: Model) -> List[Optional[Model]]: """Parses records found with model provided Parameters diff --git a/aws_lambda_powertools/utilities/parser/parser.py b/aws_lambda_powertools/utilities/parser/parser.py index 1983b54dfbb..16cdc45c907 100644 --- a/aws_lambda_powertools/utilities/parser/parser.py +++ b/aws_lambda_powertools/utilities/parser/parser.py @@ -12,7 +12,7 @@ @lambda_handler_decorator def event_parser( - handler: Callable[[Dict, Any], Any], + handler: Callable[[Any, LambdaContext], Any], event: Dict[str, Any], context: LambdaContext, model: Model, From de53605afc8e9e278c953c83b73603cc4fa9884f Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Sun, 25 Oct 2020 14:49:18 +0100 Subject: [PATCH 23/26] docs: reorder extending models as parse fn wasn't introduced Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 150 +++++++++++++++--------------- 1 file changed, 75 insertions(+), 75 deletions(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 308a991e8e8..17f84cb8e00 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -57,80 +57,6 @@ These are simply Python classes that inherit from BaseModel. **Parser** enforces Use Koudai Aono's data model code generation tool for Pydantic
-### Extending built-in models - -Parser comes with the following built-in models: - -Model name | Description -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- -**DynamoDBStreamModel** | Lambda Event Source payload for Amazon DynamoDB Streams -**EventBridgeModel** | Lambda Event Source payload for Amazon EventBridge -**SqsModel** | Lambda Event Source payload for Amazon SQS - -You can extend them to include your own models, and yet have all other known fields parsed along the way. - -**EventBridge example** - -```python:title=extending_builtin_models.py -from aws_lambda_powertools.utilities.parser import parse, BaseModel -from aws_lambda_powertools.utilities.parser.models import EventBridgeModel - -from typing import List, Optional - -class OrderItem(BaseModel): - id: int - quantity: int - description: str - -class Order(BaseModel): - id: int - description: str - items: List[OrderItem] - -# highlight-start -class OrderEventModel(EventBridgeModel): - detail: Order -# highlight-end - -payload = { - "version": "0", - "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", - "detail-type": "OrderPurchased", - "source": "OrderService", - "account": "111122223333", - "time": "2020-10-22T18:43:48Z", - "region": "us-west-1", - "resources": ["some_additional"], - "detail": { # highlight-line - "id": 10876546789, - "description": "My order", - "items": [ - { - "id": 1015938732, - "quantity": 1, - "description": "item xpto" - } - ] - } -} - -ret = parse(model=OrderEventModel, event=payload) # highlight-line - -assert ret.source == "OrderService" -assert ret.detail.description == "My order" -assert ret.detail_type == "OrderPurchased" # we rename it to snake_case since detail-type is an invalid name - -for order_item in ret.detail.items: - ... -``` - -**What's going on here, you might ask**: - -1. We imported our built-in model `EventBridgeModel` from the parser utility -2. Defined how our `Order` should look like -3. Defined how part of our EventBridge event should look like by overriding `detail` key within our `OrderEventModel` -4. Parser parsed the original event against `OrderEventModel` - ## Parsing events You can parse inbound events using **event_parser** decorator, or the standalone `parse` function. Both are also able to parse either dictionary or JSON string as an input. @@ -141,7 +67,7 @@ Use the decorator for fail fast scenarios where you want your Lambda function to `event_parser` decorator will throw a `ValidationError` if your event cannot be parsed according to the model. -```python=:title=event_parser_decorator.py +```python:title=event_parser_decorator.py from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, ValidationError from aws_lambda_powertools.utilities.typing import LambdaContext import json @@ -321,6 +247,80 @@ parse(model=UserModel, event=payload) You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in Pydantic's documentation.
+## Extending built-in models + +Parser comes with the following built-in models: + +Model name | Description +------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- +**DynamoDBStreamModel** | Lambda Event Source payload for Amazon DynamoDB Streams +**EventBridgeModel** | Lambda Event Source payload for Amazon EventBridge +**SqsModel** | Lambda Event Source payload for Amazon SQS + +You can extend them to include your own models, and yet have all other known fields parsed along the way. + +**EventBridge example** + +```python:title=extending_builtin_models.py +from aws_lambda_powertools.utilities.parser import parse, BaseModel +from aws_lambda_powertools.utilities.parser.models import EventBridgeModel + +from typing import List, Optional + +class OrderItem(BaseModel): + id: int + quantity: int + description: str + +class Order(BaseModel): + id: int + description: str + items: List[OrderItem] + +# highlight-start +class OrderEventModel(EventBridgeModel): + detail: Order +# highlight-end + +payload = { + "version": "0", + "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718", + "detail-type": "OrderPurchased", + "source": "OrderService", + "account": "111122223333", + "time": "2020-10-22T18:43:48Z", + "region": "us-west-1", + "resources": ["some_additional"], + "detail": { # highlight-line + "id": 10876546789, + "description": "My order", + "items": [ + { + "id": 1015938732, + "quantity": 1, + "description": "item xpto" + } + ] + } +} + +ret = parse(model=OrderEventModel, event=payload) # highlight-line + +assert ret.source == "OrderService" +assert ret.detail.description == "My order" +assert ret.detail_type == "OrderPurchased" # we rename it to snake_case since detail-type is an invalid name + +for order_item in ret.detail.items: + ... +``` + +**What's going on here, you might ask**: + +1. We imported our built-in model `EventBridgeModel` from the parser utility +2. Defined how our `Order` should look like +3. Defined how part of our EventBridge event should look like by overriding `detail` key within our `OrderEventModel` +4. Parser parsed the original event against `OrderEventModel` + ## Envelopes From 57681a2d885182c13e8f4fe06fbb6be481e76afe Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Sun, 25 Oct 2020 15:27:54 +0100 Subject: [PATCH 24/26] docs: reorder data validation; improve envelopes section Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 239 ++++++++++++++++-------------- 1 file changed, 130 insertions(+), 109 deletions(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 17f84cb8e00..f30e8a1b059 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -70,6 +70,7 @@ Use the decorator for fail fast scenarios where you want your Lambda function to ```python:title=event_parser_decorator.py from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, ValidationError from aws_lambda_powertools.utilities.typing import LambdaContext + import json class OrderItem(BaseModel): @@ -152,101 +153,6 @@ def my_function(): } ``` -### Data model validation - - - This is radically different from the Validator utility which validates events against JSON Schema. -
- -You can use parser's validator for deep inspection of object values and complex relationships. - -There are two types of class method decorators you can use: - -* **`validator`** - Useful to quickly validate an individual field and its value -* **`root_validator`** - Useful to validate the entire model's data - -Keep the following in mind regardless of which decorator you end up using it: - -* You must raise either `ValueError`, `TypeError`, or `AssertionError` when value is not compliant -* You must return the value(s) itself if compliant - -#### Validating fields - -Quick validation to verify whether the field `message` has the value of `hello world`. - -```python:title=deep_data_validation.py -from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator - -class HelloWorldModel(BaseModel): - message: str - - @validator('message') # highlight-line - def is_hello_world(cls, v): - if v != "hello world": - raise ValueError("Message must be hello world!") - return v - -parse(model=HelloWorldModel, event={"message": "hello universe"}) -``` - -If you run as-is, you should expect the following error with the message we provided in our exception: - -``` -message - Message must be hello world! (type=value_error) -``` - -Alternatively, you can pass `'*'` as an argument for the decorator so that you can validate every value available. - -```python:title=validate_all_field_values.py -from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator - -class HelloWorldModel(BaseModel): - message: str - sender: str - - @validator('*') # highlight-line - def has_whitespace(cls, v): - if ' ' not in v: - raise ValueError("Must have whitespace...") - - return v - -parse(model=HelloWorldModel, event={"message": "hello universe", "sender": "universe"}) -``` - -#### Validating entire model - -`root_validator` can help when you have a complex validation mechanism. For example finding whether data has been omitted, comparing field values, etc. - -```python:title=validate_all_field_values.py -from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator - -class UserModel(BaseModel): - username: str - password1: str - password2: str - - @root_validator - def check_passwords_match(cls, values): - pw1, pw2 = values.get('password1'), values.get('password2') - if pw1 is not None and pw2 is not None and pw1 != pw2: - raise ValueError('passwords do not match') - return values - -payload = { - "username": "universe", - "password1": "myp@ssword", - "password2": "repeat password" -} - -parse(model=UserModel, event=payload) -``` - - - You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in Pydantic's documentation. -
- ## Extending built-in models Parser comes with the following built-in models: @@ -324,12 +230,22 @@ for order_item in ret.detail.items: ## Envelopes -Envelope parameter is useful when your actual payload is wrapped around a known structure, for example Lambda Event Sources like EventBridge. +When trying to parse your payloads wrapped in a known structure, you might encounter the following situations: + +* Your actual payload is wrapped around a known structure, for example Lambda Event Sources like EventBridge +* You're only interested in a portion of the payload, for example parsing the `detail` of custom events in EventBridge, or `body` of SQS records + +You can either solve these situations by creating a model of these known structures, parsing them, then extracting and parsing a key where your payload is. -Example of parsing a model found in an event coming from EventBridge, where all you want is what's inside the `detail` key. +This can become difficult quite quickly. Parser makes this problem easier through a feature named `Envelope`. + +Envelopes can be used via `envelope` parameter available in both `parse` function and `event_parser` decorator. + +Here's an example of parsing a model found in an event coming from EventBridge, where all you want is what's inside the `detail` key. ```python:title=parse_eventbridge_payload.py -from aws_lambda_powertools.utilities.parser import parse, BaseModel, envelopes +from aws_lambda_powertools.utilities.parser import event_parser, parse, BaseModel, envelopes +from aws_lambda_powertools.utilities.typing import LambdaContext class UserModel(BaseModel): username: str @@ -358,6 +274,11 @@ ret = parse(model=UserModel, envelope=envelopes.EventBridgeModel, event=payload) # Parsed model only contains our actual model, not the entire EventBridge + Payload parsed assert ret.password1 == ret.password2 + +# Same behaviour but using our decorator +@event_parser(model=UserModel, envelope=envelopes.EventBridgeModel) # highlight-line +def handler(event: UserModel, context: LambdaContext): + assert event.password1 == event.password2 ``` **What's going on here, you might ask**: @@ -367,17 +288,18 @@ assert ret.password1 == ret.password2 3. Parser parsed the original event against the EventBridge model 4. Parser then parsed the `detail` key using `UserModel` + ### Built-in envelopes -Parser comes with the following built-in envelopes, where `BaseModel` in the return section is your given model. +Parser comes with the following built-in envelopes, where `Model` in the return section is your given model. Envelope name | Behaviour | Return ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------ -**DynamoDBStreamEnvelope** | 1. Parses data using `DynamoDBStreamModel`.
2. Parses records in `NewImage` and `OldImage` keys using your model.
3. Returns a list with a dictionary containing `NewImage` and `OldImage` keys | `List[Dict[Literal["NewImage", "OldImage"], BaseModel]]` -**EventBridgeEnvelope** | 1. Parses data using `EventBridgeModel`.
2. Parses `detail` key using your model and returns it. | `BaseModel` -**SqsEnvelope** | 1. Parses data using `SqsModel`.
2. Parses records in `body` key using your model and return them in a list. | `List[BaseModel]` +**DynamoDBStreamEnvelope** | 1. Parses data using `DynamoDBStreamModel`.
2. Parses records in `NewImage` and `OldImage` keys using your model.
3. Returns a list with a dictionary containing `NewImage` and `OldImage` keys | `List[Dict[str, Optional[Model]]]` +**EventBridgeEnvelope** | 1. Parses data using `EventBridgeModel`.
2. Parses `detail` key using your model and returns it. | `Model` +**SqsEnvelope** | 1. Parses data using `SqsModel`.
2. Parses records in `body` key using your model and return them in a list. | `List[Model]` -### Bringing your own envelope model +### Bringing your own envelope You can create your own Envelope model and logic by inheriting from `BaseEnvelope`, and implementing the `parse` method. @@ -407,20 +329,23 @@ class EventBridgeModel(BaseModel): **EventBridge Envelope** ```python:title=eventbridge_envelope.py -from aws_lambda_powertools.utilities.parser import BaseEnvelope, BaseModel -from typing import Any, Dict -from ..models import EventBridgeModel +from aws_lambda_powertools.utilities.parser import BaseEnvelope, models +from aws_lambda_powertools.utilities.parser.models import EventBridgeModel + +from typing import Any, Dict, Optional, TypeVar + +Model = TypeVar("Model", bound=BaseModel) class EventBridgeEnvelope(BaseEnvelope): # highlight-line - def parse(self, data: Dict[str, Any], model: BaseModel) -> BaseModel: # highlight-line + def parse(self, data: Optional[Union[Dict[str, Any], Any]], model: Model) -> Optional[Model]: # highlight-line """Parses data found with model provided Parameters ---------- data : Dict Lambda event to be parsed - model : BaseModel + model : Model Data model provided to parse after extracting data using envelope Returns @@ -428,7 +353,7 @@ class EventBridgeEnvelope(BaseEnvelope): # highlight-line Any Parsed detail payload with model provided """ - parsed_envelope = EventBridgeModel(**data) # highlight-line + parsed_envelope = EventBridgeModel.parse_obj(data) # highlight-line return self._parse(data=parsed_envelope.detail, model=model) # highlight-line ``` @@ -439,6 +364,102 @@ class EventBridgeEnvelope(BaseEnvelope): # highlight-line 3. Then, we parsed the incoming data with our envelope to confirm it matches EventBridge's structure defined in `EventBridgeModel` 4. Lastly, we call `_parse` from `BaseEnvelope` to parse the data in our envelope (.detail) using the customer model +### Data model validation + + + This is radically different from the Validator utility which validates events against JSON Schema. +
+ +You can use parser's validator for deep inspection of object values and complex relationships. + +There are two types of class method decorators you can use: + +* **`validator`** - Useful to quickly validate an individual field and its value +* **`root_validator`** - Useful to validate the entire model's data + +Keep the following in mind regardless of which decorator you end up using it: + +* You must raise either `ValueError`, `TypeError`, or `AssertionError` when value is not compliant +* You must return the value(s) itself if compliant + +#### Validating fields + +Quick validation to verify whether the field `message` has the value of `hello world`. + +```python:title=deep_data_validation.py +from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator + +class HelloWorldModel(BaseModel): + message: str + + @validator('message') # highlight-line + def is_hello_world(cls, v): + if v != "hello world": + raise ValueError("Message must be hello world!") + return v + +parse(model=HelloWorldModel, event={"message": "hello universe"}) +``` + +If you run as-is, you should expect the following error with the message we provided in our exception: + +``` +message + Message must be hello world! (type=value_error) +``` + +Alternatively, you can pass `'*'` as an argument for the decorator so that you can validate every value available. + +```python:title=validate_all_field_values.py +from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator + +class HelloWorldModel(BaseModel): + message: str + sender: str + + @validator('*') # highlight-line + def has_whitespace(cls, v): + if ' ' not in v: + raise ValueError("Must have whitespace...") + + return v + +parse(model=HelloWorldModel, event={"message": "hello universe", "sender": "universe"}) +``` + +#### Validating entire model + +`root_validator` can help when you have a complex validation mechanism. For example finding whether data has been omitted, comparing field values, etc. + +```python:title=validate_all_field_values.py +from aws_lambda_powertools.utilities.parser import parse, BaseModel, validator + +class UserModel(BaseModel): + username: str + password1: str + password2: str + + @root_validator + def check_passwords_match(cls, values): + pw1, pw2 = values.get('password1'), values.get('password2') + if pw1 is not None and pw2 is not None and pw1 != pw2: + raise ValueError('passwords do not match') + return values + +payload = { + "username": "universe", + "password1": "myp@ssword", + "password2": "repeat password" +} + +parse(model=UserModel, event=payload) +``` + + + You can read more about validating list items, reusing validators, validating raw inputs, and a lot more in Pydantic's documentation. +
+ + ## FAQ **When should I use parser vs data_classes utility?** From eeabc0fa86bca3d2484e9984e208eb596c529ad8 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Sun, 25 Oct 2020 16:02:27 +0100 Subject: [PATCH 25/26] docs: address Ran's feedback Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 59 ++++++++++++++++++++++++++----- 1 file changed, 50 insertions(+), 9 deletions(-) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index f30e8a1b059..55c82ef3341 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -48,15 +48,6 @@ class Order(BaseModel): These are simply Python classes that inherit from BaseModel. **Parser** enforces type hints declared in your model at runtime. - - - - - Looking to auto-generate models from JSON, YAML, JSON Schemas, OpenApi, etc? -

- Use Koudai Aono's data model code generation tool for Pydantic -
- ## Parsing events You can parse inbound events using **event_parser** decorator, or the standalone `parse` function. Both are also able to parse either dictionary or JSON string as an input. @@ -460,6 +451,56 @@ parse(model=UserModel, event=payload)
+## Advanced use cases + + + Looking to auto-generate models from JSON, YAML, JSON Schemas, OpenApi, etc? +

+ Use Koudai Aono's data model code generation tool for Pydantic +
+ +There are number of advanced use cases well documented in Pydantic's doc such as creating [immutable models](https://pydantic-docs.helpmanual.io/usage/models/#faux-immutability), [declaring fields with dynamic values]((https://pydantic-docs.helpmanual.io/usage/models/#field-with-dynamic-default-value)) e.g. UUID, and [helper functions to parse models from files, str](https://pydantic-docs.helpmanual.io/usage/models/#helper-functions), etc. + +Two possible unknown use cases are Models and exception' serialization. Models have methods to [export them](https://pydantic-docs.helpmanual.io/usage/exporting_models/) as `dict`, `JSON`, `JSON Schema`, and Validation exceptions can be exported as JSON. + +```python:title=serializing_models_exceptions.py +from aws_lambda_powertools.utilities import Logger +from aws_lambda_powertools.utilities.parser import parse, BaseModel, ValidationError, validator + +logger = Logger(service="user") + +class UserModel(BaseModel): + username: str + password1: str + password2: str + +payload = { + "username": "universe", + "password1": "myp@ssword", + "password2": "repeat password" +} + +def my_function(): + try: + return parse(model=UserModel, event=payload) + except ValidationError as e: + logger.exception(e.json()) # highlight-line + return { + "status_code": 400, + "message": "Invalid username" + } + +User: UserModel = my_function() +# highlight-start +user_dict = User.dict() +user_json = User.json() +user_json_schema_as_dict = User.schema() +user_json_schema_as_json = User.schema_json(indent=2) +# highlight-end +``` + +These can be quite useful when manipulating models that later need to be serialized as inputs for services like DynamoDB, EventBridge, etc. + ## FAQ **When should I use parser vs data_classes utility?** From 592cd5639066154f755a0ca8ba011cd2fe3e87b5 Mon Sep 17 00:00:00 2001 From: heitorlessa Date: Sun, 25 Oct 2020 16:05:09 +0100 Subject: [PATCH 26/26] docs: add a note that decorator will replace the event Signed-off-by: heitorlessa --- docs/content/utilities/parser.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/content/utilities/parser.mdx b/docs/content/utilities/parser.mdx index 55c82ef3341..c0eedbcc3d1 100644 --- a/docs/content/utilities/parser.mdx +++ b/docs/content/utilities/parser.mdx @@ -58,6 +58,8 @@ Use the decorator for fail fast scenarios where you want your Lambda function to `event_parser` decorator will throw a `ValidationError` if your event cannot be parsed according to the model. +> NOTE: **This decorator will replace the `event` object with the parsed model if successful**. This means you might be careful when nesting other decorators that expect `event` to be a `dict`. + ```python:title=event_parser_decorator.py from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, ValidationError from aws_lambda_powertools.utilities.typing import LambdaContext