.Net7 Minimal API running under provider.al2 runtime timeout on quick fire successive requests

[Simonl9l]

Describe the bug

Related to discussion #1424, but figure an Issue might garner more attention, since prior response there were stale and now have a reproducible sample.

In summary we have a .Net 7 Minimal API setup (serverless.AspNetCoreMinimalAPI) switched to .Net7 and using the amazon.lambda.tools to build the package using the docker image process via the -ucfb True option.

Our sense is that in .Net7 the Host/Middleware, or something related to its interaction with the provider.al2 runtime that may have changed with the newer .Net version is now possibly incompatible the .AddAWSLambdaHosting middleware.

Expected Behavior

Successive and continuous request should complete successfully

Current Behavior

Once deployed we find:

• that the default route / can be continuously quick-fired without error.
• that other requests like /calculator/add/1/2 initially succeeds but then fail after ~3-5 rapid requests with a timeout, as set by the lambda deployment (30 secs in our case).
• If the invocations are spaced out ( at ~30 second intervals) the successive requests continue to work successfully.

Reproduction Steps

We have a GitHub workflow enabled clone-able/deployable repo here with instructions here

We have used the default template samples so that we can eliminate if any of our code is introducing a side effect, and whist we had added some of that code it's commented out.

Possible Solution

We need to migrate to .Net7 such that we can use latest C# features.

Additional Information/Context

We note comments in the RuntimeSupport.Client.InvocationResponse there is special handling of the OutputStream for successive invocations, but are unsure of the implication in this scenario, and assume this in an optimization.

Of note:

• The / (default) route request in the Map does not leverage any default middleware where as the other routes to map the request on the controller do.
• Whist Net7.0 is not explicitly supported for minimal API, given the lambda.NativeAOT template, we assume basic compatibility with the provider.al2 custom runtime
• We have set <PublishAot>false</PublishAot> in the csproj, to eliminate any concern that this my be an issue.

AWS .NET SDK and/or Package version used

Amazon.Lambda.AspNetCoreServer.Hosting - 1.6.0
AWSSDK.Extensions.NETCore.Setup - 3.7.7
Amazon.Lambda.RuntimeSupport - 1.8.8
Amazon.Lambda.Core" Version - 2.1.0
Amazon.Lambda.Serialization.SystemTextJson - 2.3.1

Targeted .NET Platform

.Net 7

Operating System and version

AmazonLinux

[ashishdhingra]

Needs review with @normj, related discussion #1424.

[Simonl9l]

@ashishdhingra & @normj - Thanks - to confirm you were able to deploy and reproduce?

Not sure this is related to annotations per label?

[normj]

@Simonl9l I cloned your repo and although I didn't run it through the GitHub actions I ran same dotnet lambda package and dotnet lambda deployserverless commands and had no issues. Can you add the log output you are seeing those commands in case I'm missing something?

Is there anything in the CloudWatch logs when you get the timeout?

[Simonl9l]

@normj thanks of taking a look!

You'll note that the actual Minimal API response succeeded per the top line of the log; but you will see the timeout as highlighted.

Our assumption is that the runtime is pulling the rug out for some reason, on the basis of a timeout, but this is somehow erroneous, as the full elapsed time from the first call is only ~5 seconds. logging shows:

logging shows a 30 sec gap here...but there is no reason for the code to stall, as it's exactly the same that ran successfully the two prior iterations.

Here is the full CloudWatch log of the three successive executions to where the third fails.
log-events-viewer-result-3.csv, line 370 of the file lines up with the screen shot.

Also to confirm that you are using the /calculator/add/1/2 or similar route, not the / route?

[Simonl9l]

@normj & @ashishdhingra - sorry to bug, is there anything we can do to assist the reproduction effort?

I believe I sent Norm a direct email offline if we want to connect/screen share etc.

[Simonl9l]

@normj & @ashishdhingra

It's unfortunate that you and team have not be able to recreate this issue, that we are aware. We do though have more information.

Our hypothesis is that there is something wrong with the raw AL2 image, that only manifest when using the underpinning of the AddAWSLambdaHosting that regular raw or Annotations based Lambdas do not.

We get to this because we can build a self contained executable in both .Net 7 (and .Net 8 preview 6 - just to see) and deploy them to a .Net 6 runtime lambda, and they both work as expected, with no timeouts - on basis that the executable is called bootstrap, and more importantly the basis that the self-contained this runtime is overriding the runtime in the Dotnet 6 image.

This means we have a way forwards, and can avoid the AL2 runtime entirely. It had not occurred to us that one could do this before.

However if we switch the Lambda runtime to the AL2 runtime, using the same packages in turn it runs as expected but we get exactly the same failure observations as we did before.

Of note, if we build a self-contained .Net 6 package as above and deploy that to AL2 runtime, it also fails as observed.

Our assumptions is that when .Net 8 goes LTS, the existing .Net lambda runtime will be updated to support it, and what we have identified will likely impact few, if anyone.

However as MSFT support AOT builds of the .Net runtime, that say underpins Minimal API, this may surface for those trying to run that in the current AL2 image. Perhaps well event see Amazon Linux 2023 but then - right?

Well also probably be fine in the future with running Minimal API AOT images (when supported) in the current dotnet runtime too.

Is there anything y'all know about that we may be missing in running any self-contained or AOT'd packages in the dotnet runtime that might present a future problem here?

[dguisinger]

@normj & @ashishdhingra

I appear to be having this issue as well. I have been spinning why wheels for 3 weeks on multiple .NET projects, no matter what I do my .NET 8 lambda functions timeout (30 seconds at API gateway, 60 seconds at the end of "execution") for the most simplest of things.

I am not using AoT.
I am using AddAWSLambdaHosting(LambdaEventSource.RestApi) with API Controllers, not minimal API.
We have swagger running in our projects, the swagger UI responds fine most of the time, but we have a 0% success rate on WebAPI calls.

This is making Lambda a completely unusable platform for us. I had mentioned it in the Tracking .NET 8 support issue nearly a month ago.

[normj]

@dguisinger Is there a sample project you can share that demonstrates this behavior?