Description
Bug description
Currently, our E2E tests are very flaky (fail about 40-50% of the time) due to fixed waiting time. This happens in both metrics and tracers E2E tests.
For example, tracer.test.ts waits for 2 minutes before finishing the beforeAll() function.
Sometimes, the trace come back later than two minutes. The test() blocks will fail.
Expected Behavior
The tests should not be flaky.
Current Behavior
The tests are flaky. If the traces/metrics appear after the fixed wait time.
Possible Solution
-
Implement retry The tests should never wait for fixed amount of time. Instead, it should poll for trace or metrics and retry until it gets expected number of traces. This could be implemented with promise-retry library. See example code in "Reference" section below
-
Introduce uuid However, we cannot simply count the number of traces or metrics because there might be another test running concurrently (e.g. one for Node14, another for Node12) . The traces/metrics found may be from another test running. Thus, we need to introduce a uuid in metadata and filter base on that. You can pass UUID via environment variable like in this e2e test in Logger
If you need refactoring, take note of Logger's e2e tests and try to structure it similar to that.
Steps to Reproduce
Run E2E tests a few times. You can use this script:
repeat 20 {jest --group=e2e/PACKAGE; sleep 0.5}Environment
- Powertools version used:0.7.0
- Packaging format (Layers, npm): N/A
- AWS Lambda function runtime: 12 and 14
- Debugging logs: N/A
Related issues, RFCs
N/A
Reference
Here's an example implementation on fetching log stream with promise-retry library. It keep fetching until the log group has the atLeast number of log stream. You can adapt this with traces/metrics. (but need to check uuid)
export const fetchStreamsUntil = async (logGroupName: string, atLeast:number): Promise<string[]> => {
const retryOption = {
retries: 6,
factor: 1.5,
minTimeout: 1000,
}
return promiseRetry(async (retry: any, attemptNumber: number) => {
const streams = await fetchStreams(logGroupName)
if(streams.length < atLeast){
console.debug(`Found ${streams.length} log streams, retry until having at least ${atLeast} stream`);
retry();
}
return streams;
}, retryOption)
};