Maintenance: code performance tests in CI/CD pipelines

[saragerion]

Description of the feature request

As part of our CI/CD pipelines it would be great to be able to run performance tests over each utility's code.

Problem statement

Relying on unit tests and integration tests help increasing the confidence over the production readiness of the code that's been added.
Since we are all humans who make mistakes, additional safety nets integrated via automation can hugely help.
We currently don't have such safety net when it comes to performance.

Summary of the feature

Add performance tests in our CI/CD pipelines.

TODO (each of these should probably be distinct pull requests) :

• Agree on the most suitable profiling technique / technology (based on DX, costs, ease of usage, capabilities, ...)
• Implement benchmarks to compare the performance of the features with the one of other established libraries or methodologies
• Define acceptable performance baselines and implement performance tests
• Optimize performance when needed
• Add performance tests in most suitable pipelines (for example: on push to branch, on merge to main)

Code examples

An example of technology we can vet could be Clinic.js. I haven't used it before and I am not sure how easy it is to integrate it as part of our pipelines, but worth looking into it in my opinion.
https://clinicjs.org/

Benefits for you and the wider AWS community

More performant production code for our community.

Describe alternatives you've considered
N/A

Additional context
N/A

Related issues, RFCs

N/A

[willfarrell]

I had the same idea for Middy, unfortunately clinicjs doesn't support serverless (clinicjs/node-clinic#279). I was able to get it running with a wrapper (that added noise), but automating it was a much larger task I haven't gotten to.

[ijemmy]

@saragerion I just discussed with Heitor about this. Some work has been done on Python side. We can reuse this this.

There are two approaches:

1. https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/benchmark/README.md
   This uses SAM to deploy two Lambda functions, one with Powertool and another without (line 46-47). Then force a cold start for 20 times (line 61-72) and get the result.

2. https://github.com/awslabs/aws-lambda-powertools-python/blob/develop/tests/performance/test_high_level_imports.py This profile the initial library importing locally, we probably can't reuse this as it's Python specific.

I think 1 is aligned with what we want. I would love to have this in the same Github action that triggers E2E tests so we can. We can reuse most of the code (just plugin our NodeJS function and change the runtime).

I'm wondering if we should use CDK instead of SAM so that contributors do not have to use two tools in a single project (as we use CDK in E2E).

[saragerion]

Thanks for the suggestions @ijemmy!

The approaches adopted for the Python powertool make total sense for the language and its ecosystem (Python powertools utilities being non-modular for example).

In our case, developers choose Node.js also because of its non-blocking I/O, and because it's notoriously lightweight & fast.
Cold start is not the only concern: single code functionalities' performance is a deciding factor that guides developers' decisions on whether to adopt a library or not, and as proven by this discussion point, the AWS community is interested in data points that compare the performance of single operations against other popular alternatives:
#315 (comment)

See example of popular Node.js logger Pino:
https://github.com/pinojs/pino/blob/master/docs/benchmarks.md

For this reason, at least for the first iteration of this issue, I am more inclined with starting with method 2 and then potentially tackling method 1 in a second iteration.

[willfarrell]

For unit benchmarks pino uses fastbench, another popular one is benchmark. Neither have been updated in years :(. I know some CI tooling can already integrate with the output of benchmark. Here is a sample for benchmark (https://github.com/middyjs/middy/blob/release/3.x/packages/core/__benchmarks__/index.js).

For e2e one of your colleagues published: https://github.com/aws-samples/aws-lambda-es-module-performance-benchmark using API Gateway + artillery

[saragerion]

Thanks for sharing @willfarrell :)

For e2e one of your colleagues published: https://github.com/aws-samples/aws-lambda-es-module-performance-benchmark using API Gateway + artillery

As customer and as AWS employee I worked a lot with K6:
https://k6.io/docs/
https://github.com/saragerion/serverless-voting-app/blob/main/k6/load-test-api-get-videos.js