|
1 | 1 | # Verification - test suite
|
2 | 2 |
|
| 3 | +## Measuring conformance |
3 | 4 |
|
| 5 | +In addition to the specification documents, a test suite is being developed to |
| 6 | +aid library developers check conformance to the spec. **NOTE: The test suite |
| 7 | +is still a work in progress.** It can be found at |
| 8 | +[[https://github.com/data-apis/array-api-tests]]. |
4 | 9 |
|
5 |
| -## Measuring conformance |
| 10 | +It is important to note that while the aim of the array api test suite is to |
| 11 | +cover as much as the spec as possible, there are necessarily some aspects of |
| 12 | +the spec that are not covered by the test suite, typically because they are |
| 13 | +impossible to effectively test. Furthermore, if the test suite appears to |
| 14 | +diverge in any way from what the spec documents say, this should be considered |
| 15 | +a bug in the test suite. The specification is the ground source of truth. |
| 16 | + |
| 17 | +## Running the tests |
| 18 | + |
| 19 | +To run the tests, first clone the [test suite |
| 20 | +repo](https://github.com/data-apis/array-api-tests), and install the testing |
| 21 | +dependencies, |
| 22 | + |
| 23 | + pip install pytest hypothesis |
| 24 | + |
| 25 | +or |
| 26 | + |
| 27 | + conda install pytest hypothesis |
| 28 | + |
| 29 | +as well as the array libraries that you want to test. To run the tests, you |
| 30 | +need to specify the array library that is to be tested. There are two ways to |
| 31 | +do this. One way is to set the `ARRAY_API_TESTS_MODULE` environment variable. |
| 32 | +For example |
| 33 | + |
| 34 | + ARRAY_API_TESTS_MODULE=numpy pytest |
| 35 | + |
| 36 | +Alternately, edit the `array_api_tests/_array_module.py` file and change the |
| 37 | +line |
| 38 | + |
| 39 | +```py |
| 40 | +array_module = None |
| 41 | +``` |
| 42 | + |
| 43 | +to |
| 44 | + |
| 45 | +```py |
| 46 | +import numpy as array_module |
| 47 | +``` |
| 48 | + |
| 49 | +(replacing `numpy` with the array module namespace to be tested). |
| 50 | + |
| 51 | +In either case, the tests should be run with the `pytest` command. |
| 52 | + |
| 53 | +Aside from the two testing dependencies (`pytest` and `hypothesis`), the test |
| 54 | +suite has no dependencies. In particular, it does not depend on any specific |
| 55 | +array libraries such as NumPy. All tests are run using only the array library |
| 56 | +that is being tested, comparing results against the behavior as defined in the |
| 57 | +spec. The test suite is designed to be standalone so that it can easily be vendored. |
6 | 58 |
|
7 |
| -TODO: explain how to use the test suite to measure and report on the level of |
8 |
| -(non-)conformance. |
| 59 | +See the |
| 60 | +[README](https://github.com/data-apis/array-api-tests/blob/master/README.md) |
| 61 | +in the test suite repo for more information about how to run and interpret the |
| 62 | +test suite results. |
0 commit comments