Skip to content

test(all): switch to use GitHub strategy matrix and fix flaky tests #828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
May 10, 2022
Merged

test(all): switch to use GitHub strategy matrix and fix flaky tests #828

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
8ae5f52
test(all): extend timeout or it sometimes fail on GitHub action
ijemmy May 2, 2022
6d33329
test(tracer): increase test case timeout of tracing to 5 mins
ijemmy May 2, 2022
5426209
test: use GitHub matrix strategy instead of concurrently
ijemmy May 2, 2022
4ef4807
test: remove concurrentcy restriction
ijemmy May 2, 2022
583be88
test: fix missing jest in the environment
ijemmy May 2, 2022
8bfef37
test: fix incorrect command
ijemmy May 2, 2022
1762f3b
test: fix environment variable not passed properly
ijemmy May 2, 2022
4bf50de
test: always use Node14 when running e2e tests
ijemmy May 2, 2022
d7661a4
test: add 5 chars of uuid to avoid clashing when testname is too long…
ijemmy May 2, 2022
367c90a
test: extract example and package checks out to make jobs run faster
ijemmy May 4, 2022
4a68307
test: fix flaky test when traces' subsegements haven't been fully loa…
ijemmy May 9, 2022
5eff569
test: make common steps in workflow reusable + add a work flow to det…
ijemmy May 9, 2022
3f0ff65
test: can't avoid checkout step in GH workflow
ijemmy May 9, 2022
7290157
test: fix filename typo
ijemmy May 9, 2022
1d272d9
test: check if filepath is the issue
ijemmy May 9, 2022
dbf3f35
test: reduce number of times in flaky detection
ijemmy May 9, 2022
2dde768
test: stop using reusable workflow
ijemmy May 9, 2022
1b6e5dc
test: remove detect-flaky workflow
ijemmy May 9, 2022
da44d1b
test: remove debugging logs
ijemmy May 9, 2022
c62cd4b
test: add a detect flaky workflow and remove auto trigger on push
ijemmy May 9, 2022
bf99b36
test: remove flaky test detection workflow
ijemmy May 9, 2022
bdcf8b8
test: update package.json to remove concurrently
ijemmy May 9, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 55 additions & 40 deletions .github/workflows/run-e2e-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,48 +2,63 @@ name: run-e2e-tests
on:
workflow_dispatch: {}
jobs:
run:
#########################
# Force Github action to run only a single job at a time (based on the group name)
# This is to prevent "race-condition" in building e2e tests infrastructure
#########################
concurrency:
group: e2e-tests
example-and-package-check:
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- name: "Checkout"
uses: actions/checkout@v3
- name: "Use NodeJS 14"
uses: actions/setup-node@v3
with:
# Always use version 14 as we use TypeScript target es2020
node-version: 14
- name: "Install npm@8.x"
run: npm i -g npm@next-8
- name: "Install monorepo packages"
# This installs all the dependencies of ./packages/*
# See https://github.com/npm/cli/issues/4475 to see why --foreground-scripts
run: npm ci --foreground-scripts
- name: "Install example packages"
# Since we are not managing the cdk examples with npm workspaces we install
# the dependencies in a separate step
working-directory: ./examples/cdk
run: npm ci
- name: "Test packaging"
run: |
npm run lerna-package
cd examples/cdk
npm install ../../packages/**/dist/aws-lambda-powertools-*
npm run test
e2e-tests:
runs-on: ubuntu-latest
permissions:
id-token: write # needed to interact with GitHub's OIDC Token endpoint.
contents: read
strategy:
matrix:
version: [12, 14]
package: [logger, metrics, tracing]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Outside of the scope of this PR, but we should rename the tracer folder to "tracer" for consistency. Created an issue:
#829

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#854

steps:
- name: "Checkout"
uses: actions/checkout@v3
#########################
# Release new version
#########################
- name: "Use NodeJS 14"
uses: actions/setup-node@v3
with:
node-version: '14'
- name: Install npm@8.x
run: npm i -g npm@next-8
- name: Install monorepo packages
# This installs all the dependencies of ./packages/*
# See https://github.com/npm/cli/issues/4475 to see why --foreground-scripts
run: npm ci --foreground-scripts
- name: Install example packages
# Since we are not managing the cdk examples with npm workspaces we install
# the dependencies in a separate step
working-directory: ./examples/cdk
run: npm ci
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1.6.1
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN_TO_ASSUME }}
aws-region: eu-west-1
- name: Run integration tests
run: npm run lerna-test:e2e
- name: Test packaging
run: |
npm run lerna-package
cd examples/cdk
npm install ../../packages/**/dist/aws-lambda-powertools-*
npm run test
- name: "Checkout"
uses: actions/checkout@v3
- name: "Use NodeJS 14"
uses: actions/setup-node@v3
with:
# Always use version 14 as we use TypeScript target es2020
node-version: 14
- name: "Install npm@8.x"
run: npm i -g npm@next-8
- name: "Install monorepo packages"
# This installs all the dependencies of ./packages/*
# See https://github.com/npm/cli/issues/4475 to see why --foreground-scripts
run: npm ci --foreground-scripts
- name: "Configure AWS credentials"
uses: aws-actions/configure-aws-credentials@v1.6.1
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN_TO_ASSUME }}
aws-region: eu-west-1
- name: "Run integration tests"
run: |
RUNTIME=nodejs${{ matrix.version }}x npm run test:e2e -w packages/${{ matrix.package }}
147 changes: 1 addition & 146 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion packages/commons/tests/utils/e2eUtils.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ export const createStackWithLambdaFunction = (params: StackWithLambdaFunctionOpt
};

export const generateUniqueName = (name_prefix: string, uuid: string, runtime: string, testName: string): string =>
`${name_prefix}-${runtime}-${testName}-${uuid}`.substring(0, 64);
`${name_prefix}-${runtime}-${uuid.substring(0,5)}-${testName}`.substring(0, 64);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When test name is really long, the uuid is truncated on some resources. This results in name clashing when the same test cases are run at the same time.


export const invokeFunction = async (functionName: string, times: number = 1, invocationMode: 'PARALLEL' | 'SEQUENTIAL' = 'PARALLEL', payload: FunctionPayload = {}): Promise<InvocationLogs[]> => {
const invocationLogs: InvocationLogs[] = [];
Expand Down
5 changes: 2 additions & 3 deletions packages/logger/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"test:unit": "jest --group=unit --detectOpenHandles --coverage --verbose",
"test:e2e:nodejs12x": "RUNTIME=nodejs12x jest --group=e2e",
"test:e2e:nodejs14x": "RUNTIME=nodejs14x jest --group=e2e",
"test:e2e": "concurrently \"npm:test:e2e:nodejs12x\" \"npm:test:e2e:nodejs14x\"",
"test:e2e": "jest --group=e2e",
"watch": "jest --watch --group=unit",
"build": "tsc",
"lint": "eslint --ext .ts --fix --no-error-on-unmatched-pattern src tests",
Expand Down Expand Up @@ -47,8 +47,7 @@
"devDependencies": {
"@types/lodash.clonedeep": "^4.5.6",
"@types/lodash.merge": "^4.6.6",
"@types/lodash.pickby": "^4.6.6",
"concurrently": "^7.0.0"
"@types/lodash.pickby": "^4.6.6"
},
"files": [
"lib"
Expand Down
7 changes: 4 additions & 3 deletions packages/logger/tests/e2e/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
export const RESOURCE_NAME_PREFIX = 'Logger-E2E';
export const TEST_CASE_TIMEOUT = 20_000; // 20 seconds
export const SETUP_TIMEOUT = 300_000; // 300 seconds
export const TEARDOWN_TIMEOUT = 200_000;
export const ONE_MINUTE = 60 * 1000;
export const TEST_CASE_TIMEOUT = ONE_MINUTE;
export const SETUP_TIMEOUT = 5 * ONE_MINUTE;
export const TEARDOWN_TIMEOUT = 5 * ONE_MINUTE;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this syntax. More readable!

export const STACK_OUTPUT_LOG_GROUP = 'LogGroupName';
2 changes: 1 addition & 1 deletion packages/metrics/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"test:unit": "jest --group=unit --detectOpenHandles --coverage --verbose",
"test:e2e:nodejs12x": "RUNTIME=nodejs12x jest --group=e2e",
"test:e2e:nodejs14x": "RUNTIME=nodejs14x jest --group=e2e",
"test:e2e": "concurrently \"npm:test:e2e:nodejs12x\" \"npm:test:e2e:nodejs14x\"",
"test:e2e": "jest --group=e2e",
"watch": "jest --group=unit --watch ",
"build": "tsc",
"lint": "eslint --ext .ts --fix --no-error-on-unmatched-pattern src tests",
Expand Down
8 changes: 4 additions & 4 deletions packages/metrics/tests/e2e/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
export const RESOURCE_NAME_PREFIX = 'Metrics-E2E';
export const ONE_MINUTE = 60 * 10_00;
export const TEST_CASE_TIMEOUT = 90_000; // 90 seconds
export const SETUP_TIMEOUT = 300_000; // 300 seconds
export const TEARDOWN_TIMEOUT = 200_000;
export const ONE_MINUTE = 60 * 1000;
export const TEST_CASE_TIMEOUT = 3 * ONE_MINUTE;
export const SETUP_TIMEOUT = 5 * ONE_MINUTE;
export const TEARDOWN_TIMEOUT = 5 * ONE_MINUTE;
2 changes: 1 addition & 1 deletion packages/tracing/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
"test:unit": "jest --group=unit --detectOpenHandles --coverage --verbose",
"test:e2e:nodejs12x": "RUNTIME=nodejs12x jest --group=e2e",
"test:e2e:nodejs14x": "RUNTIME=nodejs14x jest --group=e2e",
"test:e2e": "concurrently \"npm:test:e2e:nodejs12x\" \"npm:test:e2e:nodejs14x\"",
"test:e2e": "jest --group=e2e",
"watch": "jest --watch",
"build": "tsc",
"lint": "eslint --ext .ts --fix --no-error-on-unmatched-pattern src tests",
Expand Down
Loading