From 1cbe4d1f218a6c064b44ebedaef624f72c235793 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Mon, 15 Nov 2021 15:17:09 -0500 Subject: [PATCH 01/16] docs(NODE-3753): Update testing readme file --- test/readme.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/test/readme.md b/test/readme.md index c85aa13f4c4..d8a71a4d041 100644 --- a/test/readme.md +++ b/test/readme.md @@ -1,5 +1,28 @@ # How to test +All of our test automation is powered by the [Mocha test framework](https://mochajs.org/). + +| Type of Test | Test Location | About the Tests | How to Run Tests | +| ------------ | ------------- | ---------------- | ------------- | +| Unit | `/test/unit` | The unit tests test individual units of code, typically functions. These tests do **not** interact with a real database, so mocks are used instead. | `npm run check:unit` | +| Functional | `/test/functional` | The function tests test that a given feature or piece of a feature is working as expected. These tests do **not** use mocks; instead, they interact with a real database. | `npm run check:test` | +| Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run??? | `npm run check:bench` | +| Integration | /test/integration | *Coming Soon* The integration tests test that pieces of the driver work together as expect. | `npm run check:test` | +| Manual | `/test/manual` | The manual tests are functional tests that require specialized environment setups in Evergreen. **Note**: "manual" does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no script for running all of the manual tests. Instead, you can run the appropriate script based on the specialized environment you want to use: +`npm run check:atlas` to test Atlas connectivity +`npm run check:adl` to test Atlas Data Lake +`npm run check:ocsp` to test OSCP +`npm run check:kerberos` to test Kerberos +`npm run check:tls` to test TLS +`npm run check:ldap` to test LDAP authorization +| Spec | Test input and expected results: `/test/spec`. Test runners are in `/test/functional` with the the `_spec` postfix in the test file's name. Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (aka specs)](https://github.com/mongodb/specifications). The specifications include prose (written, descriptive) tests. The drivers can choose whether to manually run the prose spec tests or automate the prose spec tests. The `test/spec` directory contains the JSON and YML files that describe the input and expected results for the prose spec tests. The test runners for these files are located in `test/functional` and `/test/unit`. +| TypeScript Types | + + + + +# LAUREN MOVE TEST INFO FROM CONTRIBUTING.md + ## Special Environments There are collections of tests that test features requiring a specialized set of environment variables to be generated and set in order to run. Look below for the section that applies to the feature you are trying to test. From 999547d59d7bfd31119d4273d72bb200ee2ce187 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Tue, 16 Nov 2021 07:38:49 -0500 Subject: [PATCH 02/16] finish testing table --- test/readme.md | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/test/readme.md b/test/readme.md index d8a71a4d041..21eda6268e5 100644 --- a/test/readme.md +++ b/test/readme.md @@ -2,21 +2,17 @@ All of our test automation is powered by the [Mocha test framework](https://mochajs.org/). +Below is a summary of the types of test automation in this repo. + | Type of Test | Test Location | About the Tests | How to Run Tests | -| ------------ | ------------- | ---------------- | ------------- | +| ------------ | ------------- | --------------- | ---------------- | | Unit | `/test/unit` | The unit tests test individual units of code, typically functions. These tests do **not** interact with a real database, so mocks are used instead. | `npm run check:unit` | | Functional | `/test/functional` | The function tests test that a given feature or piece of a feature is working as expected. These tests do **not** use mocks; instead, they interact with a real database. | `npm run check:test` | -| Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run??? | `npm run check:bench` | -| Integration | /test/integration | *Coming Soon* The integration tests test that pieces of the driver work together as expect. | `npm run check:test` | -| Manual | `/test/manual` | The manual tests are functional tests that require specialized environment setups in Evergreen. **Note**: "manual" does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no script for running all of the manual tests. Instead, you can run the appropriate script based on the specialized environment you want to use: -`npm run check:atlas` to test Atlas connectivity -`npm run check:adl` to test Atlas Data Lake -`npm run check:ocsp` to test OSCP -`npm run check:kerberos` to test Kerberos -`npm run check:tls` to test TLS -`npm run check:ldap` to test LDAP authorization -| Spec | Test input and expected results: `/test/spec`. Test runners are in `/test/functional` with the the `_spec` postfix in the test file's name. Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (aka specs)](https://github.com/mongodb/specifications). The specifications include prose (written, descriptive) tests. The drivers can choose whether to manually run the prose spec tests or automate the prose spec tests. The `test/spec` directory contains the JSON and YML files that describe the input and expected results for the prose spec tests. The test runners for these files are located in `test/functional` and `/test/unit`. -| TypeScript Types | +| Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run. They are used to measure performance. | `npm run check:bench` | +| Integration | `/test/integration` | *Coming Soon* The integration tests test that pieces of the driver work together as expect. | `npm run check:test` | +| Manual | `/test/manual` | The manual tests are functional tests that require specialized environment setups in Evergreen.
**Note**: "manual" does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no script for running all of the manual tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas connectivity
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization +| Spec | Test input and expected results: `/test/spec`.
Test runners are in `/test/functional` with the `_spec` postfix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)](https://github.com/mongodb/specifications). The specs include prose (written, descriptive) tests. The driver teams can choose whether to manually run the prose spec tests or automate the prose spec tests. The `test/spec` directory contains the JSON and YAML files that describe the input and expected results for the prose spec tests. The JSON and YAML files contain the same content just formatted differently. The developer who implements the spec tests creates the YAML file and then uses a tool to convert the file to JSON. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). +| TypeScript Definition | `/test/types` | The TypeScript definition tests verify the type definitions are correct. | `npm run check:tsd` | From db27998cc02eee641e0fb51503f74e4b46ac1031 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Tue, 16 Nov 2021 08:40:12 -0500 Subject: [PATCH 03/16] Add Evergreen info & move test info from contributing.md --- CONTRIBUTING.md | 39 +----------------- test/readme.md | 106 ++++++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 99 insertions(+), 46 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 30688a97ebe..0e224e97e08 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -56,45 +56,10 @@ We recommended these VS Code extensions: -### Running the Tests +### Automated Tests -Start a mongod standalone with our [cluster_setup.sh](test/tools/cluster_setup.sh) script: `./test/tools/cluster_setup.sh server` +This repo contains a suite of automated tests. See the [Testing README](/test/readme) for more details. -Then run the tests: `npm test` - -See [test/readme.md](test/readme.md) for more information on testing in a special environment like CSFLE or Serverless. - -### Tests FAQ - -- How can I run the tests against more than a standalone? - - You can use `test/tools/cluster_setup.sh replica_set` to start a replica set. - - If you are running more than a standalone server, make sure your `ulimit` settings are in accordance with - [MongoDB's recommendations][mongodb-ulimit]. - Changing the settings on the latest versions of macOS can be tricky. See [this article][macos-ulimt] - for tips. (You likely don't need to do the complicated maxproc steps.) - - You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific deployment: - `env MONGODB_URI=mongodb://localhost:27017 npm test` - -- How can I run just one test? - - The easiest way to run a single test is by appending `.only()` to the suite or test you want to run. - For example, you could update a test function to be `it.only(‘cool test’, function() {})`. Then - run the test using `npm run check:test` for a functional or integration test or - `npm run check:unit` for a unit test. See [Mocha's documentation][mocha-only] - for more detailed information on `.only()`. - - Another way to run a single test is to use Mocha's `grep` flag. For functional or integration tests, - run `npm run check:test -- -g 'test name'`. For unit tests, run `npm run check:unit -- -g 'test name'`. - See the [Mocha documentation][mocha-grep] for information on the `grep` flag. - -- Why are some of the tests "pending"? - - Tests that we have indicated should be skipped using `.skip()` will appear as pending in the test - results. See - [Mocha's documentation][mocha-skip] for more information. ### Commit messages diff --git a/test/readme.md b/test/readme.md index 21eda6268e5..5fdaad2f069 100644 --- a/test/readme.md +++ b/test/readme.md @@ -1,24 +1,80 @@ -# How to test +# MongoDB Node Driver Test Automation + +This repo contains a test automation suite with a variety of tests. In this readme, you'll learn +about the types of tests and how to run them. + +## Run the Tests Locally + +Start a mongod standalone with our [cluster_setup.sh](test/tools/cluster_setup.sh) script: `./test/tools/cluster_setup.sh server` + +Then run the tests: `npm test` + +Note: this will run a subset of the tests that work with the standalone server topology. + +## Run the Tests in Evergreen + +[Evergreen](https://github.com/evergreen-ci/evergreen/wiki) is a continuous integration (CI) system +we use. Evergreen builds are automatically run whenever a pull request is created or when commits are +pushed to particular branches (e.g., main, 4.0, and 3.6). + +Each Evergreen build runs the test suite against a variety of build variants that include a combination +of topologies, special environments, and operating systems. By default, commits in pull requests only run a +subset of the build variants in order to save time and resources. These builds can be individually +configured in the Evergreen UI to include more build variants. + +### Manually Kicking Off Evergreen Builds + +Occasionally, you will want to manually kick off an Evergreen build in order to debug a test failure +or to run tests against uncommitted changes. + +#### Evergreen UI +You can use the Evergreen UI to choose to rerun a task (an entire set of test automation for a given topology and environment). +Evergreen does not allow you to rerun an individual test. + +#### Evergreen CLI +You can also choose to run a build against code on your local machine that you have not yet committed +by running a pre-commit patch build. + +##### Setup + +Begin by setting up the Evergreen CLI. + +1. Download and install the Evergreen CLI according to the instructions in the [Evergreen Documentation](https://github.com/evergreen-ci/evergreen/wiki/Using-the-Command-Line-Tool). +1. Be sure to create `evergreen.yml` as described in the documentation. +1. Add the Evergreen binary to your path. + +##### Running the Build + +Once you have the Evergreen CLI setup, you are ready to run a build. + +1. In a terminal, navigate to your node driver directory: + + `cd node-mongodb-native` +1. Use the Evergreen `patch` command. `-y` skips the confirmation dialog. `-u` includes uncommitted changes. `-p [project name]` specifies the Evergreen project. --browse opens the patch URL in your browser. + + `evergreen patch -y -u -p mongo-node-driver --browse` +1. In your browser, select the build variants and tasks to run. + +## About the Tests All of our test automation is powered by the [Mocha test framework](https://mochajs.org/). +Some of the tests require a particular topology (e.g., standalone server, replica set, or sharded cluster). These tests +check the topology of the MongoDB server that is being used. If the topology does not match, the +tests will be skipped. + Below is a summary of the types of test automation in this repo. | Type of Test | Test Location | About the Tests | How to Run Tests | | ------------ | ------------- | --------------- | ---------------- | -| Unit | `/test/unit` | The unit tests test individual units of code, typically functions. These tests do **not** interact with a real database, so mocks are used instead. | `npm run check:unit` | +| Unit | `/test/unit` | The unit tests test individual pieces of code, typically functions. These tests do **not** interact with a real database, so mocks are used instead. | `npm run check:unit` | | Functional | `/test/functional` | The function tests test that a given feature or piece of a feature is working as expected. These tests do **not** use mocks; instead, they interact with a real database. | `npm run check:test` | | Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run. They are used to measure performance. | `npm run check:bench` | -| Integration | `/test/integration` | *Coming Soon* The integration tests test that pieces of the driver work together as expect. | `npm run check:test` | +| Integration | `/test/integration` | *Coming Soon* The integration tests test that pieces of the driver work together as expected. | `npm run check:test` | | Manual | `/test/manual` | The manual tests are functional tests that require specialized environment setups in Evergreen.
**Note**: "manual" does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no script for running all of the manual tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas connectivity
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization -| Spec | Test input and expected results: `/test/spec`.
Test runners are in `/test/functional` with the `_spec` postfix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)](https://github.com/mongodb/specifications). The specs include prose (written, descriptive) tests. The driver teams can choose whether to manually run the prose spec tests or automate the prose spec tests. The `test/spec` directory contains the JSON and YAML files that describe the input and expected results for the prose spec tests. The JSON and YAML files contain the same content just formatted differently. The developer who implements the spec tests creates the YAML file and then uses a tool to convert the file to JSON. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). +| Spec | Test input and expected results: `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)](https://github.com/mongodb/specifications). Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` as appropriate. For the JSON tests, a developer uses a tool to convert the JSON test file to YAML. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). | TypeScript Definition | `/test/types` | The TypeScript definition tests verify the type definitions are correct. | `npm run check:tsd` | - - - -# LAUREN MOVE TEST INFO FROM CONTRIBUTING.md - ## Special Environments There are collections of tests that test features requiring a specialized set of environment variables to be generated and set in order to run. Look below for the section that applies to the feature you are trying to test. @@ -135,3 +191,35 @@ Define the following variables in your environment: - LDAP - Snappy (maybe in general, how to test optional dependencies) - Atlas connectivity + +### Tests FAQ + +- How can I run the tests against more than a standalone? + + You can use `test/tools/cluster_setup.sh replica_set` to start a replica set. + + If you are running more than a standalone server, make sure your `ulimit` settings are in accordance with + [MongoDB's recommendations][mongodb-ulimit]. + Changing the settings on the latest versions of macOS can be tricky. See [this article][macos-ulimt] + for tips. (You likely don't need to do the complicated maxproc steps.) + + You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific deployment: + `env MONGODB_URI=mongodb://localhost:27017 npm test` + +- How can I run just one test? + + The easiest way to run a single test is by appending `.only()` to the suite or test you want to run. + For example, you could update a test function to be `it.only(‘cool test’, function() {})`. Then + run the test using `npm run check:test` for a functional or integration test or + `npm run check:unit` for a unit test. See [Mocha's documentation][mocha-only] + for more detailed information on `.only()`. + + Another way to run a single test is to use Mocha's `grep` flag. For functional or integration tests, + run `npm run check:test -- -g 'test name'`. For unit tests, run `npm run check:unit -- -g 'test name'`. + See the [Mocha documentation][mocha-grep] for information on the `grep` flag. + +- Why are some of the tests "pending"? + + Tests that we have indicated should be skipped using `.skip()` will appear as pending in the test + results. See + [Mocha's documentation][mocha-skip] for more information. From 0b9212dfff019c8fe0dbb62579a9515ad53a2c63 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Tue, 16 Nov 2021 10:25:39 -0500 Subject: [PATCH 04/16] Add topology info --- test/readme.md | 72 +++++++++++++++++++++++++++++++------------------- 1 file changed, 45 insertions(+), 27 deletions(-) diff --git a/test/readme.md b/test/readme.md index 5fdaad2f069..cfa20ed0b0b 100644 --- a/test/readme.md +++ b/test/readme.md @@ -3,14 +3,59 @@ This repo contains a test automation suite with a variety of tests. In this readme, you'll learn about the types of tests and how to run them. +## About the Tests + +All of our test automation is powered by the [Mocha test framework](https://mochajs.org/). + +Some of the tests require a particular topology (e.g., standalone server, replica set, or sharded cluster). These tests +check the topology of the MongoDB server that is being used. If the topology does not match, the +tests will be skipped. + +Below is a summary of the types of test automation in this repo. + +| Type of Test | Test Location | About the Tests | How to Run Tests | +| ------------ | ------------- | --------------- | ---------------- | +| Unit | `/test/unit` | The unit tests test individual pieces of code, typically functions. These tests do **not** interact with a real database, so mocks are used instead. | `npm run check:unit` | +| Functional | `/test/functional` | The function tests test that a given feature or piece of a feature is working as expected. These tests do **not** use mocks; instead, they interact with a real database. | `npm run check:test` | +| Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run. They are used to measure performance. | `npm run check:bench` | +| Integration | `/test/integration` | *Coming Soon* The integration tests test that pieces of the driver work together as expected. | `npm run check:test` | +| Manual | `/test/manual` | The manual tests are functional tests that require specialized environment setups in Evergreen.
**Note**: "manual" does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no script for running all of the manual tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas connectivity
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization +| Spec | Test input and expected results: `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)](https://github.com/mongodb/specifications). Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` as appropriate. For the JSON tests, a developer uses a tool to convert the JSON test file to YAML. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). +| TypeScript Definition | `/test/types` | The TypeScript definition tests verify the type definitions are correct. | `npm run check:tsd` | + + ## Run the Tests Locally +The easiest way to get started running the tests locally is to start a standalone server and run all of the tests. + Start a mongod standalone with our [cluster_setup.sh](test/tools/cluster_setup.sh) script: `./test/tools/cluster_setup.sh server` Then run the tests: `npm test` Note: this will run a subset of the tests that work with the standalone server topology. +In the following subsections, we'll dig into the details of running the tests. + +### Testing Different MongoDB Topologies + +As we mentioned earlier, the tests check the topology of the MongoDB server being used and runs the +tests associated with that topology. Tests that don't have a matching topology will be skipped. + +In the steps above, we started a standalone server: `./test/tools/cluster_setup.sh server`. + +You can use the same [cluster_setup.sh](test/tools/cluster_setup.sh) script to start a replica set +or sharded cluster by passing the appropriate option: `./test/tools/cluster_setup.sh replica_set` or +`./test/tools/cluster_setup.sh sharded_cluster`. If you are +running more than a standalone server, make sure your `ulimit` settings are in accordance with +[MongoDB's recommendations][mongodb-ulimit]. Changing the settings on the latest versions of macOS +can be tricky. See [this article][macos-ulimt] for tips. (You likely don't need to do the complicated maxproc steps.) + +You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific +deployment. For example, for a standalone server, you might use: +`env MONGODB_URI=mongodb://localhost:27017 npm test`. For a replica set, you might use: +`env MONGODB_URI=mongodb://localhost:31000,localhost:31001,localhost:31002/?replicaSet=rs npm test`. +******LAUREN STILL VERIFYING THIS****** + ## Run the Tests in Evergreen [Evergreen](https://github.com/evergreen-ci/evergreen/wiki) is a continuous integration (CI) system @@ -55,25 +100,6 @@ Once you have the Evergreen CLI setup, you are ready to run a build. `evergreen patch -y -u -p mongo-node-driver --browse` 1. In your browser, select the build variants and tasks to run. -## About the Tests - -All of our test automation is powered by the [Mocha test framework](https://mochajs.org/). - -Some of the tests require a particular topology (e.g., standalone server, replica set, or sharded cluster). These tests -check the topology of the MongoDB server that is being used. If the topology does not match, the -tests will be skipped. - -Below is a summary of the types of test automation in this repo. - -| Type of Test | Test Location | About the Tests | How to Run Tests | -| ------------ | ------------- | --------------- | ---------------- | -| Unit | `/test/unit` | The unit tests test individual pieces of code, typically functions. These tests do **not** interact with a real database, so mocks are used instead. | `npm run check:unit` | -| Functional | `/test/functional` | The function tests test that a given feature or piece of a feature is working as expected. These tests do **not** use mocks; instead, they interact with a real database. | `npm run check:test` | -| Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run. They are used to measure performance. | `npm run check:bench` | -| Integration | `/test/integration` | *Coming Soon* The integration tests test that pieces of the driver work together as expected. | `npm run check:test` | -| Manual | `/test/manual` | The manual tests are functional tests that require specialized environment setups in Evergreen.
**Note**: "manual" does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no script for running all of the manual tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas connectivity
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization -| Spec | Test input and expected results: `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)](https://github.com/mongodb/specifications). Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` as appropriate. For the JSON tests, a developer uses a tool to convert the JSON test file to YAML. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). -| TypeScript Definition | `/test/types` | The TypeScript definition tests verify the type definitions are correct. | `npm run check:tsd` | ## Special Environments @@ -196,15 +222,7 @@ Define the following variables in your environment: - How can I run the tests against more than a standalone? - You can use `test/tools/cluster_setup.sh replica_set` to start a replica set. - - If you are running more than a standalone server, make sure your `ulimit` settings are in accordance with - [MongoDB's recommendations][mongodb-ulimit]. - Changing the settings on the latest versions of macOS can be tricky. See [this article][macos-ulimt] - for tips. (You likely don't need to do the complicated maxproc steps.) - You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific deployment: - `env MONGODB_URI=mongodb://localhost:27017 npm test` - How can I run just one test? From cc07a1916136afe8a4c2b63209ad7eaf192997f3 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Tue, 16 Nov 2021 15:19:08 -0500 Subject: [PATCH 05/16] Rearranging --- CONTRIBUTING.md | 6 ----- test/readme.md | 66 ++++++++++++++++++++++++++----------------------- 2 files changed, 35 insertions(+), 37 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 0e224e97e08..ef94a6dae36 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -153,11 +153,5 @@ Reviewers should use the following questions to evaluate the implementation for [mtools-install]: http://blog.rueckstiess.com/mtools/install.html [nvm-windows]: https://github.com/coreybutler/nvm-windows#installation--upgrades [nvm-unix]: https://github.com/nvm-sh/nvm#install--update-script -[macos-ulimt]: https://wilsonmar.github.io/maximum-limits/ [workspace-file]: https://gist.githubusercontent.com/nbbeeken/d831a3801b4c463648c077b27da5057b/raw/8e986843e5e28019f7c0cebe5c6fa72407bf8afb/node-mongodb-native.code-workspace -[mongodb-ulimit]: https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings -[mocha-only]: https://mochajs.org/#exclusive-tests -[mocha-grep]: https://mochajs.org/#command-line-usage -[mocha-ulimit]: https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings -[mocha-skip]: https://mochajs.org/#inclusive-tests [node-jira]: https://jira.mongodb.org/browse/NODE diff --git a/test/readme.md b/test/readme.md index cfa20ed0b0b..d93ba7efd65 100644 --- a/test/readme.md +++ b/test/readme.md @@ -5,7 +5,7 @@ about the types of tests and how to run them. ## About the Tests -All of our test automation is powered by the [Mocha test framework](https://mochajs.org/). +All of our test automation is powered by the [Mocha test framework][mocha]. Some of the tests require a particular topology (e.g., standalone server, replica set, or sharded cluster). These tests check the topology of the MongoDB server that is being used. If the topology does not match, the @@ -20,7 +20,7 @@ Below is a summary of the types of test automation in this repo. | Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run. They are used to measure performance. | `npm run check:bench` | | Integration | `/test/integration` | *Coming Soon* The integration tests test that pieces of the driver work together as expected. | `npm run check:test` | | Manual | `/test/manual` | The manual tests are functional tests that require specialized environment setups in Evergreen.
**Note**: "manual" does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no script for running all of the manual tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas connectivity
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization -| Spec | Test input and expected results: `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)](https://github.com/mongodb/specifications). Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` as appropriate. For the JSON tests, a developer uses a tool to convert the JSON test file to YAML. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). +| Spec | Test input and expected results: `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)][driver-specs]. Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` as appropriate. For the JSON tests, a developer uses a tool to convert the JSON test file to YAML. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). | TypeScript Definition | `/test/types` | The TypeScript definition tests verify the type definitions are correct. | `npm run check:tsd` | @@ -28,11 +28,15 @@ Below is a summary of the types of test automation in this repo. The easiest way to get started running the tests locally is to start a standalone server and run all of the tests. -Start a mongod standalone with our [cluster_setup.sh](test/tools/cluster_setup.sh) script: `./test/tools/cluster_setup.sh server` +Start a mongod standalone with our [cluster_setup.sh](test/tools/cluster_setup.sh) script: `./test/tools/cluster_setup.sh server`. -Then run the tests: `npm test` +Then run the tests: `npm test`. -Note: this will run a subset of the tests that work with the standalone server topology. +**Note:** the command above will run a subset of the tests that work with the standalone server topology since the tests are being run against a standalone server. + +The output will show how many tests passed, failed, and are pending. +Tests that we have indicated should be skipped using `.skip()` will appear as pending in the test +results. See [Mocha's documentation][mocha-skip] for more information. In the following subsections, we'll dig into the details of running the tests. @@ -54,11 +58,22 @@ You can prefix `npm test` with a `MONGODB_URI` environment variable to point the deployment. For example, for a standalone server, you might use: `env MONGODB_URI=mongodb://localhost:27017 npm test`. For a replica set, you might use: `env MONGODB_URI=mongodb://localhost:31000,localhost:31001,localhost:31002/?replicaSet=rs npm test`. -******LAUREN STILL VERIFYING THIS****** + +### Running Individual Tests + +The easiest way to run a single test is by appending `.only()` to the suite or test you want to run. +For example, you could update a test function to be `it.only(‘cool test’, function() {})`. Then +run the test using `npm run check:test` for a functional or integration test or +`npm run check:unit` for a unit test. See [Mocha's documentation][mocha-only] +for more detailed information on `.only()`. + +Another way to run a single test is to use Mocha's `grep` flag. For functional or integration tests, +run `npm run check:test -- -g 'test name'`. For unit tests, run `npm run check:unit -- -g 'test name'`. +See the [Mocha documentation][mocha-grep] for information on the `grep` flag. ## Run the Tests in Evergreen -[Evergreen](https://github.com/evergreen-ci/evergreen/wiki) is a continuous integration (CI) system +[Evergreen][evergreen-wiki] is a continuous integration (CI) system we use. Evergreen builds are automatically run whenever a pull request is created or when commits are pushed to particular branches (e.g., main, 4.0, and 3.6). @@ -67,6 +82,9 @@ of topologies, special environments, and operating systems. By default, commits subset of the build variants in order to save time and resources. These builds can be individually configured in the Evergreen UI to include more build variants. +Keep in mind that if you want to run only a few tests you can append `.only()` as described in the +[section above](#running-individual-tests). + ### Manually Kicking Off Evergreen Builds Occasionally, you will want to manually kick off an Evergreen build in order to debug a test failure @@ -84,7 +102,7 @@ by running a pre-commit patch build. Begin by setting up the Evergreen CLI. -1. Download and install the Evergreen CLI according to the instructions in the [Evergreen Documentation](https://github.com/evergreen-ci/evergreen/wiki/Using-the-Command-Line-Tool). +1. Download and install the Evergreen CLI according to the instructions in the [Evergreen Documentation][evergreen-docs]. 1. Be sure to create `evergreen.yml` as described in the documentation. 1. Add the Evergreen binary to your path. @@ -218,26 +236,12 @@ Define the following variables in your environment: - Snappy (maybe in general, how to test optional dependencies) - Atlas connectivity -### Tests FAQ - -- How can I run the tests against more than a standalone? - - - -- How can I run just one test? - - The easiest way to run a single test is by appending `.only()` to the suite or test you want to run. - For example, you could update a test function to be `it.only(‘cool test’, function() {})`. Then - run the test using `npm run check:test` for a functional or integration test or - `npm run check:unit` for a unit test. See [Mocha's documentation][mocha-only] - for more detailed information on `.only()`. - - Another way to run a single test is to use Mocha's `grep` flag. For functional or integration tests, - run `npm run check:test -- -g 'test name'`. For unit tests, run `npm run check:unit -- -g 'test name'`. - See the [Mocha documentation][mocha-grep] for information on the `grep` flag. - -- Why are some of the tests "pending"? - - Tests that we have indicated should be skipped using `.skip()` will appear as pending in the test - results. See - [Mocha's documentation][mocha-skip] for more information. +[mocha]: https://mochajs.org/ +[mocha-skip]: https://mochajs.org/#inclusive-tests +[mongodb-ulimit]: https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings +[macos-ulimt]: https://wilsonmar.github.io/maximum-limits/ +[mocha-only]: https://mochajs.org/#exclusive-tests +[mocha-grep]: https://mochajs.org/#command-line-usage +[evergreen-docs]: https://github.com/evergreen-ci/evergreen/wiki/Using-the-Command-Line-Tool +[evergreen-wiki]: https://github.com/evergreen-ci/evergreen/wiki +[driver-specs]: https://github.com/mongodb/specifications From 5cc82745adc0a6b21002a255cc79ecb0ba4f78fb Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Tue, 16 Nov 2021 15:25:42 -0500 Subject: [PATCH 06/16] Add note about data dir --- test/readme.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/test/readme.md b/test/readme.md index d93ba7efd65..fbbf34be450 100644 --- a/test/readme.md +++ b/test/readme.md @@ -54,6 +54,11 @@ running more than a standalone server, make sure your `ulimit` settings are in a [MongoDB's recommendations][mongodb-ulimit]. Changing the settings on the latest versions of macOS can be tricky. See [this article][macos-ulimt] for tips. (You likely don't need to do the complicated maxproc steps.) +The [cluster_setup.sh](test/tools/cluster_setup.sh) script automatically stores the files associated +with the MongoDB server in the `data` directory. You can delete this directory if you want to ensure +you're running a clean configuration. If you delete the directory, the associated database server +will be stopped, and you will need to run the [cluster_setup.sh](test/tools/cluster_setup.sh) again. + You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific deployment. For example, for a standalone server, you might use: `env MONGODB_URI=mongodb://localhost:27017 npm test`. For a replica set, you might use: From 936c3f4fd4d235d9d26ffe489f231fb08c7ae9bd Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Wed, 17 Nov 2021 08:24:08 -0500 Subject: [PATCH 07/16] Add steps on pre-release library & manual testing --- test/readme.md | 40 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/test/readme.md b/test/readme.md index fbbf34be450..1b9e6938ffd 100644 --- a/test/readme.md +++ b/test/readme.md @@ -24,7 +24,7 @@ Below is a summary of the types of test automation in this repo. | TypeScript Definition | `/test/types` | The TypeScript definition tests verify the type definitions are correct. | `npm run check:tsd` | -## Run the Tests Locally +## Running the Tests Locally The easiest way to get started running the tests locally is to start a standalone server and run all of the tests. @@ -55,7 +55,8 @@ running more than a standalone server, make sure your `ulimit` settings are in a can be tricky. See [this article][macos-ulimt] for tips. (You likely don't need to do the complicated maxproc steps.) The [cluster_setup.sh](test/tools/cluster_setup.sh) script automatically stores the files associated -with the MongoDB server in the `data` directory. You can delete this directory if you want to ensure +with the MongoDB server in the `data` directory, which is stored at the top level of this repository. +You can delete this directory if you want to ensure you're running a clean configuration. If you delete the directory, the associated database server will be stopped, and you will need to run the [cluster_setup.sh](test/tools/cluster_setup.sh) again. @@ -76,7 +77,7 @@ Another way to run a single test is to use Mocha's `grep` flag. For functional run `npm run check:test -- -g 'test name'`. For unit tests, run `npm run check:unit -- -g 'test name'`. See the [Mocha documentation][mocha-grep] for information on the `grep` flag. -## Run the Tests in Evergreen +## Running the Tests in Evergreen [Evergreen][evergreen-wiki] is a continuous integration (CI) system we use. Evergreen builds are automatically run whenever a pull request is created or when commits are @@ -123,6 +124,37 @@ Once you have the Evergreen CLI setup, you are ready to run a build. `evergreen patch -y -u -p mongo-node-driver --browse` 1. In your browser, select the build variants and tasks to run. +## Using a Pre-Release Version of a Dependent Library + +You may want to test the driver with a pre-release version of a dependent library (e.g., [bson][js-bson]). +Follow the steps below to do so. +1. Open [package.json](../package.json) +1. Identify the line that specifies the dependency +1. Replace the version number with the commit hash of the dependent library. For example: `"bson": "e29156f7438fa77c1672fd70789d7ade9ca65061"` +1. Run `npm install` to install the dependency + +Now you can run the automated tests, run manual tests, or kick off an Evergreen build from your local +repository. + +## Manually Testing the Driver + +You may want to manually test changes you have made to the driver. The steps below will walk you +through how to create a new Node project that uses your local copy of the Node driver. You can +modify the steps to work with existing Node projects. + +1. Navigate to a new directory and create a new Node project by running `npm init` in a terminal and + working through the interactive prompts. A new file named `package.json` will be created for you. +1. In `package.json`, create a new dependency for `mongodb` that points to your local copy of the + driver. For example: + ``` + "dependencies": { + "mongodb": "/path-to-your-copy-of-the-driver-repo/node-mongodb-native" + } + ``` +1. Run `npm install` to install the dependency. +1. Create a new file that uses the driver to test your changes. See the + [MongoDB Node.js Quick Start Repo][node-quick-start] for example scripts you can use. + ## Special Environments @@ -250,3 +282,5 @@ Define the following variables in your environment: [evergreen-docs]: https://github.com/evergreen-ci/evergreen/wiki/Using-the-Command-Line-Tool [evergreen-wiki]: https://github.com/evergreen-ci/evergreen/wiki [driver-specs]: https://github.com/mongodb/specifications +[node-quick-start]: https://github.com/mongodb-developer/nodejs-quickstart +[js-bson]: https://github.com/mongodb/js-bson From c58ca99a6966d9396cae193d2fb16ddd15f33267 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Wed, 17 Nov 2021 09:50:52 -0500 Subject: [PATCH 08/16] Update LB tests & formatting --- test/readme.md | 167 +++++++++++++++++++++---------------------------- 1 file changed, 71 insertions(+), 96 deletions(-) diff --git a/test/readme.md b/test/readme.md index 1b9e6938ffd..439f6709788 100644 --- a/test/readme.md +++ b/test/readme.md @@ -3,6 +3,15 @@ This repo contains a test automation suite with a variety of tests. In this readme, you'll learn about the types of tests and how to run them. +## Table of Contents + +[About the Tests](#about-the-tests) +[Running the Tests Locally](#running-the-tests-locally) +[Running the Tests in Evergreen](#running-the-tests-in-evergreen) +[Using a Pre-Release Version of a Dependent Library](#using-a-pre-release-version-of-a-dependent-library) +[Manually Testing the Driver](#manually-testing-the-driver) +[Testing with Special Environments](#testing-with-special-environments) + ## About the Tests All of our test automation is powered by the [Mocha test framework][mocha]. @@ -34,75 +43,48 @@ Then run the tests: `npm test`. **Note:** the command above will run a subset of the tests that work with the standalone server topology since the tests are being run against a standalone server. -The output will show how many tests passed, failed, and are pending. -Tests that we have indicated should be skipped using `.skip()` will appear as pending in the test -results. See [Mocha's documentation][mocha-skip] for more information. +The output will show how many tests passed, failed, and are pending. Tests that we have indicated should be skipped using `.skip()` will appear as pending in the test results. See [Mocha's documentation][mocha-skip] for more information. In the following subsections, we'll dig into the details of running the tests. ### Testing Different MongoDB Topologies -As we mentioned earlier, the tests check the topology of the MongoDB server being used and runs the -tests associated with that topology. Tests that don't have a matching topology will be skipped. +As we mentioned earlier, the tests check the topology of the MongoDB server being used and runs the tests associated with that topology. Tests that don't have a matching topology will be skipped. In the steps above, we started a standalone server: `./test/tools/cluster_setup.sh server`. -You can use the same [cluster_setup.sh](test/tools/cluster_setup.sh) script to start a replica set -or sharded cluster by passing the appropriate option: `./test/tools/cluster_setup.sh replica_set` or -`./test/tools/cluster_setup.sh sharded_cluster`. If you are -running more than a standalone server, make sure your `ulimit` settings are in accordance with -[MongoDB's recommendations][mongodb-ulimit]. Changing the settings on the latest versions of macOS -can be tricky. See [this article][macos-ulimt] for tips. (You likely don't need to do the complicated maxproc steps.) +You can use the same [cluster_setup.sh](test/tools/cluster_setup.sh) script to start a replica set or sharded cluster by passing the appropriate option: `./test/tools/cluster_setup.sh replica_set` or +`./test/tools/cluster_setup.sh sharded_cluster`. If you are running more than a standalone server, make sure your `ulimit` settings are in accordance with [MongoDB's recommendations][mongodb-ulimit]. Changing the settings on the latest versions of macOS can be tricky. See [this article][macos-ulimt] for tips. (You likely don't need to do the complicated maxproc steps.) -The [cluster_setup.sh](test/tools/cluster_setup.sh) script automatically stores the files associated -with the MongoDB server in the `data` directory, which is stored at the top level of this repository. -You can delete this directory if you want to ensure -you're running a clean configuration. If you delete the directory, the associated database server -will be stopped, and you will need to run the [cluster_setup.sh](test/tools/cluster_setup.sh) again. +The [cluster_setup.sh](test/tools/cluster_setup.sh) script automatically stores the files associated with the MongoDB server in the `data` directory, which is stored at the top level of this repository. +You can delete this directory if you want to ensure you're running a clean configuration. If you delete the directory, the associated database server will be stopped, and you will need to run the [cluster_setup.sh](test/tools/cluster_setup.sh) again. -You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific -deployment. For example, for a standalone server, you might use: -`env MONGODB_URI=mongodb://localhost:27017 npm test`. For a replica set, you might use: -`env MONGODB_URI=mongodb://localhost:31000,localhost:31001,localhost:31002/?replicaSet=rs npm test`. +You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific deployment. For example, for a standalone server, you might use: `env MONGODB_URI=mongodb://localhost:27017 npm test`. For a replica set, you might use: `env MONGODB_URI=mongodb://localhost:31000,localhost:31001,localhost:31002/?replicaSet=rs npm test`. ### Running Individual Tests -The easiest way to run a single test is by appending `.only()` to the suite or test you want to run. -For example, you could update a test function to be `it.only(‘cool test’, function() {})`. Then -run the test using `npm run check:test` for a functional or integration test or -`npm run check:unit` for a unit test. See [Mocha's documentation][mocha-only] -for more detailed information on `.only()`. +The easiest way to run a single test is by appending `.only()` to the suite or test you want to run. For example, you could update a test function to be `it.only(‘cool test’, function() {})`. Then +run the test using `npm run check:test` for a functional or integration test or `npm run check:unit` for a unit test. See [Mocha's documentation][mocha-only] for more detailed information on `.only()`. -Another way to run a single test is to use Mocha's `grep` flag. For functional or integration tests, -run `npm run check:test -- -g 'test name'`. For unit tests, run `npm run check:unit -- -g 'test name'`. -See the [Mocha documentation][mocha-grep] for information on the `grep` flag. +Another way to run a single test is to use Mocha's `grep` flag. For functional or integration tests, run `npm run check:test -- -g 'test name'`. For unit tests, run `npm run check:unit -- -g 'test name'`. See the [Mocha documentation][mocha-grep] for information on the `grep` flag. ## Running the Tests in Evergreen -[Evergreen][evergreen-wiki] is a continuous integration (CI) system -we use. Evergreen builds are automatically run whenever a pull request is created or when commits are -pushed to particular branches (e.g., main, 4.0, and 3.6). +[Evergreen][evergreen-wiki] is a continuous integration (CI) system we use. Evergreen builds are automatically run whenever a pull request is created or when commits are pushed to particular branches (e.g., main, 4.0, and 3.6). -Each Evergreen build runs the test suite against a variety of build variants that include a combination -of topologies, special environments, and operating systems. By default, commits in pull requests only run a -subset of the build variants in order to save time and resources. These builds can be individually -configured in the Evergreen UI to include more build variants. +Each Evergreen build runs the test suite against a variety of build variants that include a combination of topologies, special environments, and operating systems. By default, commits in pull requests only run a subset of the build variants in order to save time and resources. These builds can be individually configured in the Evergreen UI to include more build variants. -Keep in mind that if you want to run only a few tests you can append `.only()` as described in the -[section above](#running-individual-tests). +Keep in mind that if you want to run only a few tests you can append `.only()` as described in the [section above on running individual tests](#running-individual-tests). ### Manually Kicking Off Evergreen Builds -Occasionally, you will want to manually kick off an Evergreen build in order to debug a test failure -or to run tests against uncommitted changes. +Occasionally, you will want to manually kick off an Evergreen build in order to debug a test failure or to run tests against uncommitted changes. #### Evergreen UI -You can use the Evergreen UI to choose to rerun a task (an entire set of test automation for a given topology and environment). -Evergreen does not allow you to rerun an individual test. +You can use the Evergreen UI to choose to rerun a task (an entire set of test automation for a given topology and environment). Evergreen does not allow you to rerun an individual test. #### Evergreen CLI -You can also choose to run a build against code on your local machine that you have not yet committed -by running a pre-commit patch build. +You can also choose to run a build against code on your local machine that you have not yet committed by running a pre-commit patch build. ##### Setup @@ -138,30 +120,32 @@ repository. ## Manually Testing the Driver -You may want to manually test changes you have made to the driver. The steps below will walk you -through how to create a new Node project that uses your local copy of the Node driver. You can +You may want to manually test changes you have made to the driver. The steps below will walk you through how to create a new Node project that uses your local copy of the Node driver. You can modify the steps to work with existing Node projects. -1. Navigate to a new directory and create a new Node project by running `npm init` in a terminal and - working through the interactive prompts. A new file named `package.json` will be created for you. -1. In `package.json`, create a new dependency for `mongodb` that points to your local copy of the - driver. For example: +1. Navigate to a new directory and create a new Node project by running `npm init` in a terminal and working through the interactive prompts. A new file named `package.json` will be created for you. +1. In `package.json`, create a new dependency for `mongodb` that points to your local copy of the driver. For example: ``` "dependencies": { "mongodb": "/path-to-your-copy-of-the-driver-repo/node-mongodb-native" } ``` 1. Run `npm install` to install the dependency. -1. Create a new file that uses the driver to test your changes. See the - [MongoDB Node.js Quick Start Repo][node-quick-start] for example scripts you can use. +1. Create a new file that uses the driver to test your changes. See the [MongoDB Node.js Quick Start Repo][node-quick-start] for example scripts you can use. + +## Testing with Special Environments -## Special Environments +In order to test some features, you will need to generate and set a specialized group of environment variables. The subsections below will walk you through how to generate and set the environment variables for these features. -There are collections of tests that test features requiring a specialized set of environment variables to be generated and set in order to run. Look below for the section that applies to the feature you are trying to test. +Before you begin any of the subsections below, clone the [drivers-evergreen-tools repo](https://github.com/mongodb-labs/drivers-evergreen-tools.git). + +We recommend creating an environment variable named `DRIVERS_TOOLS` that stores the path to your local copy of the driver-evergreen-tools repo: `export DRIVERS_TOOLS="/path/to/your/copy/of/drivers-evergreen-tools" ### Serverless +Create a new serverless instance in Atlas. + Find the following script in driver-evergreen-tools and make sure you have the following environment variables defined. _**Remember**_ some of these are sensitive credentials so keep them safe and only put them in your environment when you need them. @@ -206,51 +190,42 @@ Lastly, comment out the `source` of `install-dependencies.sh` command in `.everg ### Load Balanced -You'll first need to start a sharded cluster using your favorite MongoDB orchestration tool of choice. -The tool should create a cluster with two mongos so you have a uri like `MONGODB_URI=mongodb://host1,host2/` -Then you need to start a load balancer: you can install `haproxy` on macos via `brew` and use the script provided in drivers-evergreen-tools. - -```sh -$DRIVERS_TOOLS/.evergreen/run-load-balancer.sh start -``` - -This will output an evergreen expansion file: `lb-expansion.yml` +The following steps will walk you through how to start and test a load balancer. -```yaml -SINGLE_MONGOS_LB_URI: "mongodb://127.0.0.1:8000/?loadBalanced=true" -MULTI_MONGOS_LB_URI: "mongodb://127.0.0.1:8001/?loadBalanced=true" -``` - -Since it's a flat yaml file, you can run the following to get a sourceable environment file: - -```sh -cat lb-expansion.yml | sed 's/: /=/g' > lb.env -``` - -You have to add an additional environment variable to the end of this `lb.env` file: - -```sh -FAKE_MONGODB_SERVICE_ID="true" -``` - -This enables logic in the driver to stick in a fake service id on responses since that's what a real LB deployment is required to do. -With those variables sourced, you can run the whole test suite as you normally would. - -> Please note, `FAKE_MONGODB_SERVICE_ID` will no longer be needed with the completion of [NODE-3431](https://jira.mongodb.org/browse/NODE-3431). - -```sh -npm run check:test -``` - -Take note of the `[ topology type: load-balanced ]` printout from mocha to make sure it picked up the environment as expected. - -When you are done testing you can shutdown the haproxy load balancer with: - -```sh -$DRIVERS_TOOLS/.evergreen/run-load-balancer.sh stop -``` +1. Start a sharded cluster. You can use the [cluster_setup.sh](test/tools/cluster_setup.sh) script to do so: `./test/tools/cluster_setup.sh sharded_cluster`. The tool should create a cluster with two mongos, so you have a URI similar to `MONGODB_URI=mongodb://host1,host2/`. +1. Create an environment variable named `MONGODB_URI` that stores the URI of the sharded cluster you just created. For example: `export MONGODB_URI="mongodb://host1,host2/"` +1. Install the HAProxy load balancer. For those on macOS, you can install HAProxy with `brew install haproxy`. +1. Start the load balancer by using the [run-load-balancer script](https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/run-load-balancer.sh) provided in drivers-evergreen-tools. + ```sh + $DRIVERS_TOOLS/.evergreen/run-load-balancer.sh start + ``` + A new file name `lb-expansion.yml` will be automatically created. The contents of the file will be similar in structure to the code below. + ```yaml + SINGLE_MONGOS_LB_URI: "mongodb://127.0.0.1:8000/?loadBalanced=true" + MULTI_MONGOS_LB_URI: "mongodb://127.0.0.1:8001/?loadBalanced=true" + ``` +1. Generate a sourceable environment file from `lb-expansion.yml` by running the following command: + ```sh + cat lb-expansion.yml | sed 's/: /=/g' > lb.env + ``` + A new file name `lb.env` is automatically created. +1. Add an additional environment variable named `FAKE_MONGODB_SERVICE_ID` to the end of the `lb.env` file. Setting `FAKE_MONGODB_SERVICE_ID` to `true` enables logic in the driver to stick in a fake service ID on responses since that's what a real load balanced deployment is required to do. + ```sh + FAKE_MONGODB_SERVICE_ID="true" + ``` + > Please note, `FAKE_MONGODB_SERVICE_ID` will no longer be needed with the completion of [NODE-3431](https://jira.mongodb.org/browse/NODE-3431). +1. Source the environment variables using a command like `source lb.env` +1. Run the test suite as you normally would: + ```sh + npm run check:test + ``` + Take note of the `[ topology type: load-balanced ]` printout from mocha to make sure it picked up the environment as expected. +1. When you are done testing, shutdown the HAProxy load balancer: + ```sh + $DRIVERS_TOOLS/.evergreen/run-load-balancer.sh stop + ``` -### CSFLE +### Client-Side Field Level Encryption (CSFLE) As long as certain environment variables are present and mongodb-client-encryption is installed, FLE will run with a regular mocha execution: `npm run check:test`. From abfc5238fc1e5e3eade2d3f8acf3f12555544f02 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Wed, 17 Nov 2021 09:52:01 -0500 Subject: [PATCH 09/16] Fix table of contents --- test/readme.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/test/readme.md b/test/readme.md index 439f6709788..05d7cb87c2d 100644 --- a/test/readme.md +++ b/test/readme.md @@ -5,12 +5,12 @@ about the types of tests and how to run them. ## Table of Contents -[About the Tests](#about-the-tests) -[Running the Tests Locally](#running-the-tests-locally) -[Running the Tests in Evergreen](#running-the-tests-in-evergreen) -[Using a Pre-Release Version of a Dependent Library](#using-a-pre-release-version-of-a-dependent-library) -[Manually Testing the Driver](#manually-testing-the-driver) -[Testing with Special Environments](#testing-with-special-environments) +- [About the Tests](#about-the-tests) +- [Running the Tests Locally](#running-the-tests-locally) +- [Running the Tests in Evergreen](#running-the-tests-in-evergreen) +- [Using a Pre-Release Version of a Dependent Library](#using-a-pre-release-version-of-a-dependent-library) +- [Manually Testing the Driver](#manually-testing-the-driver) +- [Testing with Special Environments](#testing-with-special-environments) ## About the Tests From c1b84da126b069e4be08840ad5439bb00dd042c1 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Wed, 17 Nov 2021 09:56:10 -0500 Subject: [PATCH 10/16] Fix link --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ef94a6dae36..84d734d7fe0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -58,7 +58,7 @@ We recommended these VS Code extensions: ### Automated Tests -This repo contains a suite of automated tests. See the [Testing README](/test/readme) for more details. +This repo contains a suite of automated tests. See the [Testing README](/test/readme.md) for more details. ### Commit messages From 0f6af49c3ce357e7c82a4afa7c7d706119516b9c Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Wed, 17 Nov 2021 15:19:21 -0500 Subject: [PATCH 11/16] Add notes for LB tests --- test/readme.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/test/readme.md b/test/readme.md index 05d7cb87c2d..8839f24c097 100644 --- a/test/readme.md +++ b/test/readme.md @@ -28,7 +28,7 @@ Below is a summary of the types of test automation in this repo. | Functional | `/test/functional` | The function tests test that a given feature or piece of a feature is working as expected. These tests do **not** use mocks; instead, they interact with a real database. | `npm run check:test` | | Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run. They are used to measure performance. | `npm run check:bench` | | Integration | `/test/integration` | *Coming Soon* The integration tests test that pieces of the driver work together as expected. | `npm run check:test` | -| Manual | `/test/manual` | The manual tests are functional tests that require specialized environment setups in Evergreen.
**Note**: "manual" does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no script for running all of the manual tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas connectivity
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization +| Specialized Environment | `/test/manual` | The specalized environment tests are functional tests that require specialized environment setups in Evergreen.
**Note**: "manual" in the directory path does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no single script for running all of the specialized environment tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas connectivity
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization | Spec | Test input and expected results: `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)][driver-specs]. Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` as appropriate. For the JSON tests, a developer uses a tool to convert the JSON test file to YAML. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). | TypeScript Definition | `/test/types` | The TypeScript definition tests verify the type definitions are correct. | `npm run check:tsd` | @@ -214,12 +214,13 @@ The following steps will walk you through how to start and test a load balancer. FAKE_MONGODB_SERVICE_ID="true" ``` > Please note, `FAKE_MONGODB_SERVICE_ID` will no longer be needed with the completion of [NODE-3431](https://jira.mongodb.org/browse/NODE-3431). -1. Source the environment variables using a command like `source lb.env` +1. Source the environment variables using a command like `source lb.env`. +1. Export **each** of the environment variables that were created in `lb.env`. For example: `export SINGLE_MONGOS_LB_URI`. 1. Run the test suite as you normally would: ```sh npm run check:test ``` - Take note of the `[ topology type: load-balanced ]` printout from mocha to make sure it picked up the environment as expected. + Verify that the output from Mocha includes `[ topology type: load-balanced ]`. This indicates the tests successfully accessed the specialized environment variables for load balancer testing. 1. When you are done testing, shutdown the HAProxy load balancer: ```sh $DRIVERS_TOOLS/.evergreen/run-load-balancer.sh stop From fb49e88f6c2be49a32a8f3def61bd7a2199b1605 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Thu, 18 Nov 2021 09:51:01 -0500 Subject: [PATCH 12/16] Update Specialized Environment sections --- test/readme.md | 114 ++++++++++++++++++++++++++++++------------------- 1 file changed, 71 insertions(+), 43 deletions(-) diff --git a/test/readme.md b/test/readme.md index 8839f24c097..25ff687f650 100644 --- a/test/readme.md +++ b/test/readme.md @@ -41,7 +41,7 @@ Start a mongod standalone with our [cluster_setup.sh](test/tools/cluster_setup.s Then run the tests: `npm test`. -**Note:** the command above will run a subset of the tests that work with the standalone server topology since the tests are being run against a standalone server. +>**Note:** the command above will run a subset of the tests that work with the standalone server topology since the tests are being run against a standalone server. The output will show how many tests passed, failed, and are pending. Tests that we have indicated should be skipped using `.skip()` will appear as pending in the test results. See [Mocha's documentation][mocha-skip] for more information. @@ -138,55 +138,69 @@ modify the steps to work with existing Node projects. In order to test some features, you will need to generate and set a specialized group of environment variables. The subsections below will walk you through how to generate and set the environment variables for these features. +We recommend using a different terminal for each specialized environment to avoid the environment variables from one test impacting the test runs of another type. + Before you begin any of the subsections below, clone the [drivers-evergreen-tools repo](https://github.com/mongodb-labs/drivers-evergreen-tools.git). -We recommend creating an environment variable named `DRIVERS_TOOLS` that stores the path to your local copy of the driver-evergreen-tools repo: `export DRIVERS_TOOLS="/path/to/your/copy/of/drivers-evergreen-tools" +We recommend creating an environment variable named `DRIVERS_TOOLS` that stores the path to your local copy of the driver-evergreen-tools repo: `export DRIVERS_TOOLS="/path/to/your/copy/of/drivers-evergreen-tools"`. ### Serverless -Create a new serverless instance in Atlas. +The following steps will walk you through how to create and test a MongoDB Serverless instance. + +1. Create the following environment variables using a command like `export PROJECT="node-driver"`. -Find the following script in driver-evergreen-tools and make sure you have the following environment variables defined. -_**Remember**_ some of these are sensitive credentials so keep them safe and only put them in your environment when you need them. + - `PROJECT` + - `SERVERLESS_DRIVERS_GROUP` + - `SERVERLESS_API_PUBLIC_KEY` + - `SERVERLESS_API_PRIVATE_KEY` + - `SERVERLESS_ATLAS_USER` + - `SERVERLESS_ATLAS_PASSWORD` + - `LOADBALANCED` -- `PROJECT` -- `SERVERLESS_DRIVERS_GROUP` -- `SERVERLESS_API_PUBLIC_KEY` -- `SERVERLESS_API_PRIVATE_KEY` -- `SERVERLESS_ATLAS_USER` -- `SERVERLESS_ATLAS_PASSWORD` -- `LOADBALANCED` + TODO: Explain what these variables represent and how to get their values. -```sh -$DRIVERS_TOOLS/.evergreen/serverless/create-instance.sh -``` + _**Remember**_ some of these are sensitive credentials so keep them safe and only put them in your environment when you need them. + +1. Run the [create-instance][create-instance-script] script: + ```sh + $DRIVERS_TOOLS/.evergreen/serverless/create-instance.sh + ``` + The script will take a few minutes to run. When it is finished, a new file named `serverless-expansion.yml` will be created in the current working directory. The file will contain information about an Evergreen expansion: + + ```yml + MONGODB_URI: xxx + MONGODB_SRV_URI: xxx + SERVERLESS_INSTANCE_NAME: xxx + SSL: xxx + AUTH: xxx + TOPOLOGY: xxx + SERVERLESS: xxx + MULTI_ATLASPROXY_SERVERLESS_URI: xxx + SINGLE_ATLASPROXY_SERVERLESS_URI: xxx + ``` -this will output an evergreen expansion in `serverless-expansion.yml` in the current working directory. +1. Generate a sourceable environment file from `serverless-expansion.yml` by running the following command: -```yml -MONGODB_URI: xxx -MONGODB_SRV_URI: xxx -SERVERLESS_INSTANCE_NAME: xxx -SSL: xxx -AUTH: xxx -TOPOLOGY: xxx -SERVERLESS: xxx -MULTI_ATLASPROXY_SERVERLESS_URI: xxx -SINGLE_ATLASPROXY_SERVERLESS_URI: xxx -``` + ```sh + cat serverless-expansion.yml | sed 's/: /=/g' > serverless.env + ``` + A new file named `serverless.env` is automatically created. -Since it's a flat yaml file, you can run the following to get a sourceable environment file: +1. Update the following variables in `serverless.env`, so that they are equivalent to what our Evergreen builds do: + - Change `MONGODB_URI` to have the same value as `SINGLE_ATLASPROXY_SERVERLESS_URI`. + - Add `SINGLE_MONGOS_LB_URI` and set it to the value of `SINGLE_ATLASPROXY_SERVERLESS_URI`. + - Add `MULTI_MONGOS_LB_URI` and set it to the value of `SINGLE_ATLASPROXY_SERVERLESS_URI`. -```sh -cat serverless-expansion.yml | sed 's/: /=/g' > serverless.env -``` +1. Source the environment variables using a command like `source serverless.env`. -Before sourcing `serverless.env`, make some adjustments that are equivalent to what our EVG does: +1. Export **each** of the environment variables that were created in `serverless.env`. For example: `export SINGLE_MONGOS_LB_URI`. -- Change `MONGODB_URI` to be the same as `SINGLE_ATLASPROXY_SERVERLESS_URI` -- Add `SINGLE_MONGOS_LB_URI` and `MULTI_MONGOS_LB_URI` and set them to `SINGLE_ATLASPROXY_SERVERLESS_URI` +1. Comment out the line in `.evergreen/run-serverless-tests.sh` that sources `install-dependencies.sh`. -Lastly, comment out the `source` of `install-dependencies.sh` command in `.evergreen/run-serverless-tests.sh` and you can run that script directly to test serverless instances from your local machine. +1. Run the `.evergreen/run-serverless-tests.sh` script directly to test serverless instances from your local machine. + +> Hint: If the test script fails with an error along the lines of `Uncaught TypeError: Cannot read properties of undefined (reading 'processId')`, ensure you do **not** have the `FAKE_MONGODB_SERVICE_ID` environment variable set. ### Load Balanced @@ -213,7 +227,7 @@ The following steps will walk you through how to start and test a load balancer. ```sh FAKE_MONGODB_SERVICE_ID="true" ``` - > Please note, `FAKE_MONGODB_SERVICE_ID` will no longer be needed with the completion of [NODE-3431](https://jira.mongodb.org/browse/NODE-3431). + > **Note:** `FAKE_MONGODB_SERVICE_ID` will no longer be needed with the completion of [NODE-3431](https://jira.mongodb.org/browse/NODE-3431). 1. Source the environment variables using a command like `source lb.env`. 1. Export **each** of the environment variables that were created in `lb.env`. For example: `export SINGLE_MONGOS_LB_URI`. 1. Run the test suite as you normally would: @@ -228,15 +242,27 @@ The following steps will walk you through how to start and test a load balancer. ### Client-Side Field Level Encryption (CSFLE) -As long as certain environment variables are present and mongodb-client-encryption is installed, FLE will run with a regular mocha execution: `npm run check:test`. +The following steps will walk you through how to run the tests for CSFLE. + +1. Install [MongoDB Client Encryption][npm-csfle] if you haven't already: + `npm install mongodb-client-encryption` + +1. Create the following environment variables using a command like `export AWS_REGION="us-east-1"`. + + - `AWS_ACCESS_KEY_ID` + - `AWS_SECRET_ACCESS_KEY` + - `CSFLE_KMS_PROVIDERS` + - `AWS_REGION` + - `AWS_CMK_ID` + + TODO: Explain what these variables represent and how to get their values. + +1. Run the functional tests: + + `npm run check:test` -Define the following variables in your environment: + The output of the tests will include sections like "Client Side Encryption Corpus," "Client Side Encryption Functional," "Client Side Encryption Prose Tests," and "Client Side Encryption." -- `AWS_ACCESS_KEY_ID` -- `AWS_SECRET_ACCESS_KEY` -- `CSFLE_KMS_PROVIDERS` -- `AWS_REGION` -- `AWS_CMK_ID` ### TODO Special Env Sections @@ -260,3 +286,5 @@ Define the following variables in your environment: [driver-specs]: https://github.com/mongodb/specifications [node-quick-start]: https://github.com/mongodb-developer/nodejs-quickstart [js-bson]: https://github.com/mongodb/js-bson +[create-instance-script]: https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/serverless/create-instance.sh +[npm-csfle]: (https://www.npmjs.com/package/mongodb-client-encryption) From b0f3ddadb808641847a0ce97c12b058b692f81e7 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Thu, 18 Nov 2021 10:08:37 -0500 Subject: [PATCH 13/16] Cleaning --- test/readme.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/test/readme.md b/test/readme.md index 25ff687f650..055b95cbece 100644 --- a/test/readme.md +++ b/test/readme.md @@ -25,11 +25,11 @@ Below is a summary of the types of test automation in this repo. | Type of Test | Test Location | About the Tests | How to Run Tests | | ------------ | ------------- | --------------- | ---------------- | | Unit | `/test/unit` | The unit tests test individual pieces of code, typically functions. These tests do **not** interact with a real database, so mocks are used instead. | `npm run check:unit` | -| Functional | `/test/functional` | The function tests test that a given feature or piece of a feature is working as expected. These tests do **not** use mocks; instead, they interact with a real database. | `npm run check:test` | +| Functional | `/test/functional` | The functional tests test that a given feature or piece of a feature is working as expected. These tests do **not** use mocks; instead, they interact with a real database. | `npm run check:test` | | Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run. They are used to measure performance. | `npm run check:bench` | | Integration | `/test/integration` | *Coming Soon* The integration tests test that pieces of the driver work together as expected. | `npm run check:test` | -| Specialized Environment | `/test/manual` | The specalized environment tests are functional tests that require specialized environment setups in Evergreen.
**Note**: "manual" in the directory path does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no single script for running all of the specialized environment tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas connectivity
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization -| Spec | Test input and expected results: `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)][driver-specs]. Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` as appropriate. For the JSON tests, a developer uses a tool to convert the JSON test file to YAML. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). +| Specialized Environment | `/test/manual` | The specalized environment tests are functional tests that require specialized environment setups in Evergreen.

**Note**: "manual" in the directory path does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no single script for running all of the specialized environment tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization +| Spec | Test input and expected results are in `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)][driver-specs]. Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` directory as appropriate. For the JSON tests, a developer uses a tool to convert the JSON test file to YAML. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). | TypeScript Definition | `/test/types` | The TypeScript definition tests verify the type definitions are correct. | `npm run check:tsd` | @@ -49,7 +49,7 @@ In the following subsections, we'll dig into the details of running the tests. ### Testing Different MongoDB Topologies -As we mentioned earlier, the tests check the topology of the MongoDB server being used and runs the tests associated with that topology. Tests that don't have a matching topology will be skipped. +As we mentioned earlier, the tests check the topology of the MongoDB server being used and run the tests associated with that topology. Tests that don't have a matching topology will be skipped. In the steps above, we started a standalone server: `./test/tools/cluster_setup.sh server`. @@ -57,7 +57,7 @@ You can use the same [cluster_setup.sh](test/tools/cluster_setup.sh) script to s `./test/tools/cluster_setup.sh sharded_cluster`. If you are running more than a standalone server, make sure your `ulimit` settings are in accordance with [MongoDB's recommendations][mongodb-ulimit]. Changing the settings on the latest versions of macOS can be tricky. See [this article][macos-ulimt] for tips. (You likely don't need to do the complicated maxproc steps.) The [cluster_setup.sh](test/tools/cluster_setup.sh) script automatically stores the files associated with the MongoDB server in the `data` directory, which is stored at the top level of this repository. -You can delete this directory if you want to ensure you're running a clean configuration. If you delete the directory, the associated database server will be stopped, and you will need to run the [cluster_setup.sh](test/tools/cluster_setup.sh) again. +You can delete this directory if you want to ensure you're running a clean configuration. If you delete the directory, the associated database server will be stopped, and you will need to run [cluster_setup.sh](test/tools/cluster_setup.sh) again. You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific deployment. For example, for a standalone server, you might use: `env MONGODB_URI=mongodb://localhost:27017 npm test`. For a replica set, you might use: `env MONGODB_URI=mongodb://localhost:31000,localhost:31001,localhost:31002/?replicaSet=rs npm test`. @@ -70,7 +70,7 @@ Another way to run a single test is to use Mocha's `grep` flag. For functional ## Running the Tests in Evergreen -[Evergreen][evergreen-wiki] is a continuous integration (CI) system we use. Evergreen builds are automatically run whenever a pull request is created or when commits are pushed to particular branches (e.g., main, 4.0, and 3.6). +[Evergreen][evergreen-wiki] is the continuous integration (CI) system we use. Evergreen builds are automatically run whenever a pull request is created or when commits are pushed to particular branches (e.g., main, 4.0, and 3.6). Each Evergreen build runs the test suite against a variety of build variants that include a combination of topologies, special environments, and operating systems. By default, commits in pull requests only run a subset of the build variants in order to save time and resources. These builds can be individually configured in the Evergreen UI to include more build variants. @@ -138,7 +138,7 @@ modify the steps to work with existing Node projects. In order to test some features, you will need to generate and set a specialized group of environment variables. The subsections below will walk you through how to generate and set the environment variables for these features. -We recommend using a different terminal for each specialized environment to avoid the environment variables from one test impacting the test runs of another type. +We recommend using a different terminal for each specialized environment to avoid the environment variables from one specialized environment impacting the test runs for another specialized environment. Before you begin any of the subsections below, clone the [drivers-evergreen-tools repo](https://github.com/mongodb-labs/drivers-evergreen-tools.git). @@ -160,7 +160,7 @@ The following steps will walk you through how to create and test a MongoDB Serve TODO: Explain what these variables represent and how to get their values. - _**Remember**_ some of these are sensitive credentials so keep them safe and only put them in your environment when you need them. + _**Remember**_ some of these are sensitive credentials, so keep them safe and only put them in your environment when you need them. 1. Run the [create-instance][create-instance-script] script: ```sh From 6dfa078168d37b8925abe261d9d5a409a70af42b Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Thu, 18 Nov 2021 10:12:05 -0500 Subject: [PATCH 14/16] Fix cluster setup links --- test/readme.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/test/readme.md b/test/readme.md index 055b95cbece..42e13389510 100644 --- a/test/readme.md +++ b/test/readme.md @@ -37,7 +37,7 @@ Below is a summary of the types of test automation in this repo. The easiest way to get started running the tests locally is to start a standalone server and run all of the tests. -Start a mongod standalone with our [cluster_setup.sh](test/tools/cluster_setup.sh) script: `./test/tools/cluster_setup.sh server`. +Start a mongod standalone with our [cluster_setup.sh](tools/cluster_setup.sh) script: `./test/tools/cluster_setup.sh server`. Then run the tests: `npm test`. @@ -53,11 +53,11 @@ As we mentioned earlier, the tests check the topology of the MongoDB server bein In the steps above, we started a standalone server: `./test/tools/cluster_setup.sh server`. -You can use the same [cluster_setup.sh](test/tools/cluster_setup.sh) script to start a replica set or sharded cluster by passing the appropriate option: `./test/tools/cluster_setup.sh replica_set` or +You can use the same [cluster_setup.sh](tools/cluster_setup.sh) script to start a replica set or sharded cluster by passing the appropriate option: `./test/tools/cluster_setup.sh replica_set` or `./test/tools/cluster_setup.sh sharded_cluster`. If you are running more than a standalone server, make sure your `ulimit` settings are in accordance with [MongoDB's recommendations][mongodb-ulimit]. Changing the settings on the latest versions of macOS can be tricky. See [this article][macos-ulimt] for tips. (You likely don't need to do the complicated maxproc steps.) -The [cluster_setup.sh](test/tools/cluster_setup.sh) script automatically stores the files associated with the MongoDB server in the `data` directory, which is stored at the top level of this repository. -You can delete this directory if you want to ensure you're running a clean configuration. If you delete the directory, the associated database server will be stopped, and you will need to run [cluster_setup.sh](test/tools/cluster_setup.sh) again. +The [cluster_setup.sh](tools/cluster_setup.sh) script automatically stores the files associated with the MongoDB server in the `data` directory, which is stored at the top level of this repository. +You can delete this directory if you want to ensure you're running a clean configuration. If you delete the directory, the associated database server will be stopped, and you will need to run [cluster_setup.sh](tools/cluster_setup.sh) again. You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific deployment. For example, for a standalone server, you might use: `env MONGODB_URI=mongodb://localhost:27017 npm test`. For a replica set, you might use: `env MONGODB_URI=mongodb://localhost:31000,localhost:31001,localhost:31002/?replicaSet=rs npm test`. @@ -206,7 +206,7 @@ The following steps will walk you through how to create and test a MongoDB Serve The following steps will walk you through how to start and test a load balancer. -1. Start a sharded cluster. You can use the [cluster_setup.sh](test/tools/cluster_setup.sh) script to do so: `./test/tools/cluster_setup.sh sharded_cluster`. The tool should create a cluster with two mongos, so you have a URI similar to `MONGODB_URI=mongodb://host1,host2/`. +1. Start a sharded cluster. You can use the [cluster_setup.sh](tools/cluster_setup.sh) script to do so: `./test/tools/cluster_setup.sh sharded_cluster`. The tool should create a cluster with two mongos, so you have a URI similar to `MONGODB_URI=mongodb://host1,host2/`. 1. Create an environment variable named `MONGODB_URI` that stores the URI of the sharded cluster you just created. For example: `export MONGODB_URI="mongodb://host1,host2/"` 1. Install the HAProxy load balancer. For those on macOS, you can install HAProxy with `brew install haproxy`. 1. Start the load balancer by using the [run-load-balancer script](https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/run-load-balancer.sh) provided in drivers-evergreen-tools. From 685291a4461e80487f819c6038c728f21ad10a91 Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Fri, 19 Nov 2021 12:10:48 -0500 Subject: [PATCH 15/16] Updates after code review --- test/readme.md | 77 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 54 insertions(+), 23 deletions(-) diff --git a/test/readme.md b/test/readme.md index 42e13389510..756778517ce 100644 --- a/test/readme.md +++ b/test/readme.md @@ -28,8 +28,8 @@ Below is a summary of the types of test automation in this repo. | Functional | `/test/functional` | The functional tests test that a given feature or piece of a feature is working as expected. These tests do **not** use mocks; instead, they interact with a real database. | `npm run check:test` | | Benchmark | `/test/benchmarks` | The benchmark tests report how long a designated set of tests take to run. They are used to measure performance. | `npm run check:bench` | | Integration | `/test/integration` | *Coming Soon* The integration tests test that pieces of the driver work together as expected. | `npm run check:test` | -| Specialized Environment | `/test/manual` | The specalized environment tests are functional tests that require specialized environment setups in Evergreen.

**Note**: "manual" in the directory path does not refer to tests that should be run manually. These tests are automated. These tests require manual configuration in Evergreen. | There is no single script for running all of the specialized environment tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization -| Spec | Test input and expected results are in `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)][driver-specs]. Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` directory as appropriate. For the JSON tests, a developer uses a tool to convert the JSON test file to YAML. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). +| Specialized Environment | `/test/manual` | The specalized environment tests are functional tests that require specialized environment setups in Evergreen.

**Note**: "manual" in the directory path does not refer to tests that should be run manually. These tests are automated. These tests have a special Evergreen configuration and run in isolation from the other tests. | There is no single script for running all of the specialized environment tests. Instead, you can run the appropriate script based on the specialized environment you want to use:
- `npm run check:atlas` to test Atlas
- `npm run check:adl` to test Atlas Data Lake
- `npm run check:ocsp` to test OSCP
- `npm run check:kerberos` to test Kerberos
- `npm run check:tls` to test TLS
- `npm run check:ldap` to test LDAP authorization +| Spec | Test input and expected results are in `/test/spec`.
Test runners are in `/test/functional` with the `_spec` suffix in the test file's name.
Some spec tests are also in `/test/unit`. | All of the MongoDB drivers follow the same [specifications (specs)][driver-specs]. Each spec has tests associated with it. Some of the tests are prose (written, descriptive) tests. Other tests are written in YAML and converted to JSON. The developers on the driver teams automate these tests. For prose tests, the tests are converted to automation and stored in the `test/unit` or `test/functional` directory as appropriate. Both the JSON and YAML files are stored in `test/spec`. The test runners for the JSON and YAML files are located in `test/functional` and `/test/unit`. | `npm run check:test` to run all of the functional and integration tests (including the spec tests stored with those). `npm run check:unit` to run all of the unit tests (including the spec tests stored with those). | TypeScript Definition | `/test/types` | The TypeScript definition tests verify the type definitions are correct. | `npm run check:tsd` | @@ -59,11 +59,11 @@ You can use the same [cluster_setup.sh](tools/cluster_setup.sh) script to start The [cluster_setup.sh](tools/cluster_setup.sh) script automatically stores the files associated with the MongoDB server in the `data` directory, which is stored at the top level of this repository. You can delete this directory if you want to ensure you're running a clean configuration. If you delete the directory, the associated database server will be stopped, and you will need to run [cluster_setup.sh](tools/cluster_setup.sh) again. -You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific deployment. For example, for a standalone server, you might use: `env MONGODB_URI=mongodb://localhost:27017 npm test`. For a replica set, you might use: `env MONGODB_URI=mongodb://localhost:31000,localhost:31001,localhost:31002/?replicaSet=rs npm test`. +You can prefix `npm test` with a `MONGODB_URI` environment variable to point the tests to a specific deployment. For example, for a standalone server, you might use: `MONGODB_URI=mongodb://localhost:27017 npm test`. For a replica set, you might use: `MONGODB_URI=mongodb://localhost:31000,localhost:31001,localhost:31002/?replicaSet=rs npm test`. ### Running Individual Tests -The easiest way to run a single test is by appending `.only()` to the suite or test you want to run. For example, you could update a test function to be `it.only(‘cool test’, function() {})`. Then +The easiest way to run a single test is by appending `.only()` to the test context you want to run. For example, you could update a test function to be `it.only(‘cool test’, function() {})`. Then run the test using `npm run check:test` for a functional or integration test or `npm run check:unit` for a unit test. See [Mocha's documentation][mocha-only] for more detailed information on `.only()`. Another way to run a single test is to use Mocha's `grep` flag. For functional or integration tests, run `npm run check:test -- -g 'test name'`. For unit tests, run `npm run check:unit -- -g 'test name'`. See the [Mocha documentation][mocha-grep] for information on the `grep` flag. @@ -72,9 +72,7 @@ Another way to run a single test is to use Mocha's `grep` flag. For functional [Evergreen][evergreen-wiki] is the continuous integration (CI) system we use. Evergreen builds are automatically run whenever a pull request is created or when commits are pushed to particular branches (e.g., main, 4.0, and 3.6). -Each Evergreen build runs the test suite against a variety of build variants that include a combination of topologies, special environments, and operating systems. By default, commits in pull requests only run a subset of the build variants in order to save time and resources. These builds can be individually configured in the Evergreen UI to include more build variants. - -Keep in mind that if you want to run only a few tests you can append `.only()` as described in the [section above on running individual tests](#running-individual-tests). +Each Evergreen build runs the test suite against a variety of build variants that include a combination of topologies, special environments, and operating systems. By default, commits in pull requests only run a subset of the build variants in order to save time and resources. To configure a build, update `.evergreen/config.yml.in` and then generate a new Evergreen config via `node .evergreen/generate_evergreen_tasks.js`. ### Manually Kicking Off Evergreen Builds @@ -96,7 +94,7 @@ Begin by setting up the Evergreen CLI. ##### Running the Build -Once you have the Evergreen CLI setup, you are ready to run a build. +Once you have the Evergreen CLI setup, you are ready to run a build. Keep in mind that if you want to run only a few tests, you can append `.only()` as described in the [section above on running individual tests](#running-individual-tests). 1. In a terminal, navigate to your node driver directory: @@ -112,7 +110,7 @@ You may want to test the driver with a pre-release version of a dependent librar Follow the steps below to do so. 1. Open [package.json](../package.json) 1. Identify the line that specifies the dependency -1. Replace the version number with the commit hash of the dependent library. For example: `"bson": "e29156f7438fa77c1672fd70789d7ade9ca65061"` +1. Replace the version number with the commit hash of the dependent library. For example, you could use a particular commit for the [js-bson][js-bson] project on GitHub: `"bson": "mongodb/js-bson#e29156f7438fa77c1672fd70789d7ade9ca65061"` 1. Run `npm install` to install the dependency Now you can run the automated tests, run manual tests, or kick off an Evergreen build from your local @@ -133,6 +131,8 @@ modify the steps to work with existing Node projects. 1. Run `npm install` to install the dependency. 1. Create a new file that uses the driver to test your changes. See the [MongoDB Node.js Quick Start Repo][node-quick-start] for example scripts you can use. +> **Note:** When making driver changes, you will need to run `npm run build:ts` with each change in order for it to take effect. + ## Testing with Special Environments @@ -150,15 +150,16 @@ The following steps will walk you through how to create and test a MongoDB Serve 1. Create the following environment variables using a command like `export PROJECT="node-driver"`. - - `PROJECT` - - `SERVERLESS_DRIVERS_GROUP` - - `SERVERLESS_API_PUBLIC_KEY` - - `SERVERLESS_API_PRIVATE_KEY` - - `SERVERLESS_ATLAS_USER` - - `SERVERLESS_ATLAS_PASSWORD` - - `LOADBALANCED` + > Note: MongoDB employees can pull these values from the Evergreen project's configuration. - TODO: Explain what these variables represent and how to get their values. + | Variable Name | Description | + | ------------- | ----------- | + | Project | The name of the Evergreen project where the tests will be run (e.g., `mongo-node-driver-next`) | + | `SERVERLESS_DRIVERS_GROUP` | The Atlas organization where you will be creating the serverless instance | + | `SERVERLESS_API_PUBLIC_KEY` | The [Atlas API Public Key][atlas-api-key] for the organization where you will be creating a serverless instance | + | `SERVERLESS_API_PRIVATE_KEY` | The [Atlas API Private Key][atlas-api-key] for the organization where you will be creating a serverless instance | + | `SERVERLESS_ATLAS_USER` | The [SCRAM username][scram-auth] for the Atlas user who has permission to create a serverless instance | + | `SERVERLESS_ATLAS_PASSWORD` | The [SCRAM password][scram-auth] for the Atlas user who has permission to create a serverless instance | _**Remember**_ some of these are sensitive credentials, so keep them safe and only put them in your environment when you need them. @@ -249,13 +250,41 @@ The following steps will walk you through how to run the tests for CSFLE. 1. Create the following environment variables using a command like `export AWS_REGION="us-east-1"`. - - `AWS_ACCESS_KEY_ID` - - `AWS_SECRET_ACCESS_KEY` - - `CSFLE_KMS_PROVIDERS` - - `AWS_REGION` - - `AWS_CMK_ID` + > Note: MongoDB employees can pull these values from the Evergreen project's configuration. + + | Variable Name | Description | + | ------------- | ----------- | + | `AWS_ACCESS_KEY_ID` | The AWS access key ID used to generate KMS messages | + | `AWS_SECRET_ACCESS_KEY` | The AWS secret access key used to generate KMS messages | + | `AWS_REGION` | The AWS region where the KMS resides (e.g., `us-east-1`) | + | `AWS_CMK_ID` | The Customer Master Key for the KMS | + | `CSFLE_KMS_PROVIDERS` | The raw EJSON description of the KMS providers. An example of the format is provided below. | + + The value of the `CSFLE_KMS_PROVIDERS` variable will have the following format: + + ``` + interface CSFLE_kms_providers { + aws: { + accessKeyId: string; + secretAccessKey: string; + }; + azure: { + tenantId: string; + clientId: string; + clientSecret: string; + }; + gcp: { + email: string; + privateKey: string; + }; + local: { + // EJSON handle converting this, its actually the canonical -> { $binary: { base64: string; subType: string } } + // **NOTE**: The dollar sign has to be escaped when using this as an ENV variable + key: Binary; + } + } + ``` - TODO: Explain what these variables represent and how to get their values. 1. Run the functional tests: @@ -288,3 +317,5 @@ The following steps will walk you through how to run the tests for CSFLE. [js-bson]: https://github.com/mongodb/js-bson [create-instance-script]: https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/serverless/create-instance.sh [npm-csfle]: (https://www.npmjs.com/package/mongodb-client-encryption) +[atlas-api-key]: (https://docs.atlas.mongodb.com/tutorial/configure-api-access/organization/create-one-api-key/) +[scram-auth]: (https://docs.atlas.mongodb.com/security-add-mongodb-users/#database-user-authentication) From c28150e66250b0baea77ffc9fff01ba7ec47401f Mon Sep 17 00:00:00 2001 From: Lauren Schaefer Date: Fri, 19 Nov 2021 12:12:46 -0500 Subject: [PATCH 16/16] Typos --- test/readme.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/test/readme.md b/test/readme.md index 756778517ce..24ef1e09835 100644 --- a/test/readme.md +++ b/test/readme.md @@ -154,7 +154,7 @@ The following steps will walk you through how to create and test a MongoDB Serve | Variable Name | Description | | ------------- | ----------- | - | Project | The name of the Evergreen project where the tests will be run (e.g., `mongo-node-driver-next`) | + | `Project` | The name of the Evergreen project where the tests will be run (e.g., `mongo-node-driver-next`) | | `SERVERLESS_DRIVERS_GROUP` | The Atlas organization where you will be creating the serverless instance | | `SERVERLESS_API_PUBLIC_KEY` | The [Atlas API Public Key][atlas-api-key] for the organization where you will be creating a serverless instance | | `SERVERLESS_API_PRIVATE_KEY` | The [Atlas API Private Key][atlas-api-key] for the organization where you will be creating a serverless instance | @@ -316,6 +316,6 @@ The following steps will walk you through how to run the tests for CSFLE. [node-quick-start]: https://github.com/mongodb-developer/nodejs-quickstart [js-bson]: https://github.com/mongodb/js-bson [create-instance-script]: https://github.com/mongodb-labs/drivers-evergreen-tools/blob/master/.evergreen/serverless/create-instance.sh -[npm-csfle]: (https://www.npmjs.com/package/mongodb-client-encryption) -[atlas-api-key]: (https://docs.atlas.mongodb.com/tutorial/configure-api-access/organization/create-one-api-key/) -[scram-auth]: (https://docs.atlas.mongodb.com/security-add-mongodb-users/#database-user-authentication) +[npm-csfle]: https://www.npmjs.com/package/mongodb-client-encryption +[atlas-api-key]: https://docs.atlas.mongodb.com/tutorial/configure-api-access/organization/create-one-api-key +[scram-auth]: https://docs.atlas.mongodb.com/security-add-mongodb-users/#database-user-authentication