-
Notifications
You must be signed in to change notification settings - Fork 156
chore: run e2e tests on demand #441
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 44 commits
c3b427d
27bfa90
cd7b015
7015f64
d1acd31
b367d80
52a20d8
c0a4301
38fdb75
ea45fdf
bdf800f
e3a83ce
6db56fa
6046c08
08e8050
ff9a87f
13103b8
3a189ba
5fbdc66
f86e54f
065e538
29ed2e7
0cfb133
92f1183
e196ae1
9c1e339
33d87f7
19c10c4
f792e5c
ba91ab7
51677d8
9f9b6d6
d0f9736
530d50d
df4e51a
5ff1931
9d0daa8
1d22541
1e4c897
ee00ffc
57c3bb0
0de1d03
64304ce
4f62c1a
973a9d5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,7 +7,6 @@ on: | |
jobs: | ||
publish: | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: "Checkout" | ||
uses: actions/checkout@v2 | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
name: run-e2e-tests | ||
on: | ||
workflow_dispatch: {} | ||
jobs: | ||
run: | ||
runs-on: ubuntu-latest | ||
permissions: | ||
id-token: write # needed to interact with GitHub's OIDC Token endpoint. | ||
contents: read | ||
steps: | ||
- name: "Checkout" | ||
uses: actions/checkout@v2 | ||
######################### | ||
# Release new version | ||
######################### | ||
- name: "Use NodeJS 14" | ||
uses: actions/setup-node@v2 | ||
with: | ||
node-version: '14' | ||
- name: Install packages | ||
run: | | ||
npm ci | ||
npm run lerna-ci | ||
- name: Configure AWS credentials | ||
uses: aws-actions/configure-aws-credentials@v1.6.0 | ||
with: | ||
role-to-assume: ${{ secrets.AWS_ROLE_ARN_TO_ASSUME }} | ||
aws-region: eu-west-1 | ||
- name: Run integration tests | ||
run: npm run lerna-test:e2e |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -57,6 +57,96 @@ You can build and start a local docs website by running these two commands. | |
- `npm run docs-buildDockerImage` OR `docker build -t squidfunk/mkdocs-material ./docs/` | ||
- `npm run docs-runLocalDocker` OR `docker run --rm -it -p 8000:8000 -v ${PWD}:/docs squidfunk/mkdocs-material` | ||
|
||
### Tests | ||
|
||
Tests are under `tests` folder of each modules and split into two categories: unit tests and e2e (end-to-end) tests. | ||
|
||
You can run each group separately or all together thanks to [jest-runner-groups](https://www.npmjs.com/package/jest-runner-groups). | ||
|
||
Unit tests, under `tests/unit` folder are standard [Jest](https://jestjs.io) tests. | ||
|
||
End-to-end tests, under `tests/e2e` folder, will test the module features by deploying AWS Lambda functions into your AWS Account. We use [aws-cdk](https://docs.aws.amazon.com/cdk/v1/guide/getting_started.html) v1 library (not v2 due to [this cdk issue](https://github.com/aws/aws-cdk/issues/18211)) for Typescript for creating infrastructure, and [aws-sdk-js v2](https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/) for invoking the functions and assert on the expected behaviors. All steps are also executed by Jest. | ||
|
||
|
||
Running end-to-end tests will deploy AWS resources. You will need an AWS account and the tests might incur costs. The cost from **some services** are covered by the [AWS Free Tier](https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&all-free-tier.sort-order=asc&awsf.Free%20Tier%20Types=*all&awsf.Free%20Tier%20Categories=*all) but not all of them. If you don't have an AWS Account follow [these instructions to create one](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/). | ||
|
||
When contributing to this repository, these end-to-end tests are run by the maintainers before merging a PR. | ||
|
||
|
||
**Unit tests** | ||
|
||
**Write** | ||
|
||
As mentioned before, tests are split into groups thanks to [jest-runner-groups](https://www.npmjs.com/package/jest-runner-groups) and therefore each test needs to be tagged properly by adding the following comments in the header of your unit test file: | ||
|
||
```typescript | ||
/** | ||
* Tests metrics | ||
* | ||
* @group unit/<YOUR CATEGORY>/<YOUR SUB CATEGORY> | ||
*/ | ||
``` | ||
|
||
**Run** | ||
|
||
To run unit tests you can either use | ||
* npm task `lerna-test:unit` (`npm run lerna-test:unit`) in root folder to run them all | ||
* npm task `test:e2e` (`npm run test:unit`) in module folder (for example: `packages/metrics`) to run the module specific one | ||
* jest directly `npx jest --group=unit` in module folder to run the module specific one (You can run selective tests by restricting the group to the one you want. For instance `npx jest --group=unit/metrics/class`) | ||
|
||
**e2e tests** | ||
|
||
**Write** | ||
|
||
As mentioned in the previous section, tests are split into groups thanks to [jest-runner-groups](https://www.npmjs.com/package/jest-runner-groups) and therefore each test needs to be tagged properly by adding the following comments in the header of your unit test file: | ||
|
||
```typescript | ||
/** | ||
* Tests data lake catalog | ||
* | ||
* @group e2e/<YOUR CATEGORY>/<YOUR SUB CATEGORY> | ||
*/ | ||
``` | ||
|
||
See `metrics/tests/e2e/decorator.test.ts` as an example. | ||
|
||
|
||
**Run** | ||
|
||
To run e2e tests you can either use | ||
* npm task `lerna-test:e2e` (`npm run lerna-test:e2e`) in root folder | ||
* npm task `test:e2e` (`npm run test:e2e`) in module folder (for example: `packages/metrics`) to run the module specific one | ||
* jest directly `npx jest --group=e2e` in module folder. (You can run selective tests by restricting the group to the one you want. For instance `npx jest --group=e2e/metrics/decorator`) | ||
|
||
Two important env variable can be used: | ||
saragerion marked this conversation as resolved.
Show resolved
Hide resolved
|
||
* `AWS_PROFILE` to use the right AWS credentials | ||
* `DISABLE_TEARDOWN` if you don't want your stack to be destroyed at the end of the test (useful in dev mode when iterating over your code). | ||
|
||
Example: `DISABLE_TEARDOWN=true AWS_PROFILE=dev-account npx jest --group=e2e/metrics/decorator` | ||
|
||
**Automate** | ||
|
||
flochaz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
You can run the end-to-end tests automatically on your forked project by following these steps: | ||
1. Create an IAM role in your target AWS account, with the least amount of privilege. | ||
|
||
As mentioned above in this page, we are leveraging CDK to deploy and consequently clean-up resources on AWS. Therefore to run those tests through Github actions you will need to grant specific permissions to your workflow. | ||
|
||
We recommend following [Amazon IAM best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) for the AWS credentials used in GitHub Actions workflows, including: | ||
* Do not store credentials in your repository's code. | ||
* [Grant least privilege](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege) to the credentials used in GitHub Actions workflows. Grant only the permissions required to perform the actions in your GitHub Actions workflows. | ||
* [Monitor the activity](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#keep-a-log) of the credentials used in GitHub Actions workflows. | ||
|
||
For an example of how to create a role in CDK, you can look at [@pahud/cdk-github-oidc](https://constructs.dev/packages/@pahud/cdk-github-oidc) construct. | ||
|
||
More information about: | ||
|
||
- [Github OpenID Connect](https://github.blog/changelog/2021-10-27-github-actions-secure-cloud-deployments-with-openid-connect/ | ||
- ["Configure AWS Credentials" Action For GitHub Actions](https://github.com/aws-actions/configure-aws-credentials/) | ||
1. Add your new role into your [Github fork secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) with name `AWS_ROLE_ARN_TO_ASSUME`. | ||
1. In your forked repository, go to the "Actions" tabs, select the `run-e2e-tests` workflow. | ||
1. In the run-e2e-tests workflow page, select "Run workflow" and run it on the desired branch. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: steps numbers are all "1." There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. They rendered correctly by GitHub preview & according to spec it's not required for them to be in order:
But for those who are reading the source I agree that it was confusing, changed it in 973a9d5. |
||
|
||
flochaz marked this conversation as resolved.
Show resolved
Hide resolved
|
||
> :Warning: **Don't automatically run end-to-end tests on branch push or PRs**. A malicious attacker can submit a pull request to attack your AWS account. Ideally, use a blank account without any important workload/data, and limit `AWS_ROLE_ARN_TO_ASSUME` permission to least minimum privilege. | ||
### Conventions | ||
|
||
Category | Convention | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,7 +11,9 @@ | |
}, | ||
"scripts": { | ||
"commit": "commit", | ||
"test": "jest --detectOpenHandles --coverage --verbose", | ||
"test": "npm run test:unit", | ||
"test:unit": "jest --group=unit --detectOpenHandles --coverage --verbose", | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Love it! |
||
"test:e2e": "echo 'Not Applicable'", | ||
"watch": "jest --watch", | ||
"build": "tsc", | ||
"lint": "eslint --ext .ts --fix --no-error-on-unmatched-pattern src tests", | ||
|
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Uh oh!
There was an error while loading. Please reload this page.