Bug: API Gateway Console Test button fails validation with @event_parser because requestContext -> identity -> sourceIp value is not a valid IPv4 or IPv6 network

[mikelane]

Expected Behaviour

We should be able to generate a test form the API Gateway console that allows the ApiGatewayProxyEventModel to be validated and parsed by Pydantic.

Current Behaviour

When generating a test from the API Gateway console, the value for the sourceIp in requestContext -> identity is "test-invoke-source-ip". Since this value is defined to be IPvAnyNetwork here, the validation fails in the event_parser decorator here. And so test events from the API Gateway console fail with a 502 error because of the resulting pydantic ValidationError.

Code snippet

Create a lambda handler with an event parser like this...

from aws_lambda_powertools.utilities.parser import event_parser
from aws_lambda_powertools.utilities.parser.models import APIGatewayProxyEventModel
from aws_lambda_powertools.utilities.typing import LambdaContext

@logger.inject_lambda_context(log_event=settings.debug)
@event_parser(model=APIGatewayProxyEventModel)
def handler(event: APIGatewayProxyEvent, context: LambdaContext) -> dict[str, Any]:
    logger.info('event: %s', event.json())
    return {'statusCode': 200, 'body': json.dumps({'wow': 42})}

Use the following message to the lambda:

{
    "level": "INFO",
    "location": "decorate:352",
    "message": {
        "resource": "/v1/redacted/12345",
        "path": "/v1/redacted/12345",
        "httpMethod": "GET",
        "headers": {
            "Accept": "application/json"
        },
        "multiValueHeaders": {
            "Accept": [
                "application/json"
            ]
        },
        "queryStringParameters": null,
        "multiValueQueryStringParameters": null,
        "pathParameters": {
            "redacted_id": "12345"
        },
        "stageVariables": null,
        "requestContext": {
            "resourceId": "redacted",
            "resourcePath": "/v1/redacted/{redacted_id}",
            "httpMethod": "GET",
            "extendedRequestId": "Redacted=",
            "requestTime": "30/Sep/2022:19:19:03 +0000",
            "path": "/v1/redacted/{redacted_id}",
            "accountId": "012345678912",
            "protocol": "HTTP/1.1",
            "stage": "test-invoke-stage",
            "domainPrefix": "testPrefix",
            "requestTimeEpoch": 1664565543376,
            "requestId": "9c4368e8-8510-42f1-875b-ecb63f3f5c5a",
            "identity": {
                "cognitoIdentityPoolId": null,
                "cognitoIdentityId": null,
                "apiKey": "test-invoke-api-key",
                "principalOrgId": null,
                "cognitoAuthenticationType": null,
                "userArn": "arn:aws:sts::012345678912:assumed-role/redacted/joe.bagadonuts",
                "apiKeyId": "test-invoke-api-key-id",
                "userAgent": "aws-internal/3 aws-sdk-java/1.12.302 Linux/5.4.207-126.363.amzn2int.x86_64 OpenJDK_64-Bit_Server_VM/25.342-b07 java/1.8.0_342 vendor/Oracle_Corporation cfg/retry-mode/standard",
                "accountId": "012345678912",
                "caller": "REDACTED:joe.bagadonuts",
                "sourceIp": "test-invoke-source-ip",
                "accessKey": "REDACTED",
                "cognitoAuthenticationProvider": null,
                "user": "REDACTED:joe.bagadonuts"
            },
            "domainName": "testPrefix.testDomainName",
            "apiId": "1234ASFD"
        },
        "body": null,
        "isBase64Encoded": false
    },
    "timestamp": "2022-09-30 19:19:03,390+0000",
    "service": "redacted-redacto",
    "cold_start": false,
    "function_name": "redacted",
    "function_memory_size": "128",
    "function_arn": "arn:aws:lambda:us-east-1:012345678912:function:redacted",
    "function_request_id": "bcf14ad4-c46f-471c-8b9f-60e56932c9e6",
    "xray_trace_id": "9-12345678-1234asdf2345qwer"
}

Note the error that the sourceIp is not an IPv4 or IPv6 value.

Possible Solution

Update the sourceIp here from this:

    sourceIp: IPvAnyNetwork

to this:

    sourceIp: Union[IPvAnyNetwork, Literal['test-invoke-source-ip']]

You may want to add a test (maybe in this file?) that tests for the 'test-invoke-source-ip' value explicitly.

Note this may also be an issue with the APIGatewayProxyEventV2Model because of the sourceIp definition here.

Steps to Reproduce

Create the lambda function provided above. Deploy the lambda function provided above behind an api gateway. Use the api gateway console to test the function passing in the redacted_id of 12345, and Accept:application/json for the headers.

AWS Lambda Powertools for Python version

latest

AWS Lambda function runtime

3.9

Packaging format used

PyPi

Debugging logs

No response

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[heitorlessa]

Thank you @mikelane for flagging this issue in the API Gateway service.

Would you be able to create a support case to suggest API Gateway team to use an actual IP Address in the test event?

We'll cut an internal ticket to them next week.

I'm removing the bug label as Parser is behaving as expected. That said, we should fix this at the source, otherwise this will happen again next time this changes to another literal we have no control over.

If helpful, we keep track of real events for each event source we support here: https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/tests/events/

Have a great weekend!

[heitorlessa]

Quick update: We pinged the API Gateway team asking to fix the event in the Console; we'll revert once we hear back from them.

[mikelane]

I heard back from the API Gateway support team. They said this:

"test-invoke-source-ip" is the expected result when using the API-GW test feature and unfortunately this isn't something which can be changed.
My recommendation would be to add a line of code in your Lambda as the event comes in, and then check for "test-invoke-source-ip" and then change it to a valid source IP before the validation is done.

Someone needs to budge here 😅

[heitorlessa]

Leave with us @mikelane. We reached out to the engineering team and will revert with an actionable plan from either side to get this resolved.

This can be changed on API Gateway side. Question here is whether they have the bandwidth to do it.

Worst case we disable all ip validation if that can't be honoured by the product team.

Early to tell but issues like these make me reconsider our thoughts of merging Parser/Pydantic and Event Source Data Classes - I like the value Pydantic brings, but it puts us as the weakest link in contract situations like this

[mikelane]

The fix I've suggested will work (it does work, this is what I've done to work around the issue) and would, I think, be a better solution than disabling all ip validation.