Skip to content

Update the document on the test suite #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 27, 2020
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 57 additions & 3 deletions spec/verification_test_suite.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,62 @@
# Verification - test suite

## Measuring conformance

In addition to the specification documents, a test suite is being developed to
aid library developers check conformance to the spec. **NOTE: The test suite
is still a work in progress.** It can be found at
<https://github.com/data-apis/array-api-tests>.

## Measuring conformance
It is important to note that while the aim of the array API test suite is to
cover as much of the spec as possible, there are necessarily some aspects of
the spec that are not covered by the test suite, typically because they are
impossible to effectively test. Furthermore, if the test suite appears to
diverge in any way from what the spec documents say, this should be considered
a bug in the test suite. The specification is the ground source of truth.

## Running the tests

To run the tests, first clone the [test suite
repo](https://github.com/data-apis/array-api-tests), and install the testing
dependencies,

pip install pytest hypothesis

or

conda install pytest hypothesis

as well as the array libraries that you want to test. To run the tests, you
need to specify the array library that is to be tested. There are two ways to
do this. One way is to set the `ARRAY_API_TESTS_MODULE` environment variable.
For example

ARRAY_API_TESTS_MODULE=numpy pytest

Alternatively, edit the `array_api_tests/_array_module.py` file and change the
line

```py
array_module = None
```

to

```py
import numpy as array_module
```

(replacing `numpy` with the array module namespace to be tested).

In either case, the tests should be run with the `pytest` command.

Aside from the two testing dependencies (`pytest` and `hypothesis`), the test
suite has no dependencies. In particular, it does not depend on any specific
array libraries such as NumPy. All tests are run using only the array library
that is being tested, comparing results against the behavior as defined in the
spec. The test suite is designed to be standalone so that it can easily be vendored.

TODO: explain how to use the test suite to measure and report on the level of
(non-)conformance.
See the
[README](https://github.com/data-apis/array-api-tests/blob/master/README.md)
in the test suite repo for more information about how to run and interpret the
test suite results.