From 8ffa5c02db8560e4d8e7ebdefb0f8361a225e297 Mon Sep 17 00:00:00 2001 From: Aaron Meurer Date: Mon, 26 Oct 2020 15:53:53 -0600 Subject: [PATCH 1/3] Update the document on the test suite --- spec/verification_test_suite.md | 60 +++++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 3 deletions(-) diff --git a/spec/verification_test_suite.md b/spec/verification_test_suite.md index ece853e4f..d00963a5d 100644 --- a/spec/verification_test_suite.md +++ b/spec/verification_test_suite.md @@ -1,8 +1,62 @@ # Verification - test suite +## Measuring conformance +In addition to the specification documents, a test suite is being developed to +aid library developers check conformance to the spec. **NOTE: The test suite +is still a work in progress.** It can be found at +[[https://github.com/data-apis/array-api-tests]]. -## Measuring conformance +It is important to note that while the aim of the array api test suite is to +cover as much as the spec as possible, there are necessarily some aspects of +the spec that are not covered by the test suite, typically because they are +impossible to effectively test. Furthermore, if the test suite appears to +diverge in any way from what the spec documents say, this should be considered +a bug in the test suite. The specification is the ground source of truth. + +## Running the tests + +To run the tests, first clone the [test suite +repo](https://github.com/data-apis/array-api-tests), and install the testing +dependencies, + + pip install pytest hypothesis + +or + + conda install pytest hypothesis + +as well as the array libraries that you want to test. To run the tests, you +need to specify the array library that is to be tested. There are two ways to +do this. One way is to set the `ARRAY_API_TESTS_MODULE` environment variable. +For example + + ARRAY_API_TESTS_MODULE=numpy pytest + +Alternately, edit the `array_api_tests/_array_module.py` file and change the +line + +```py +array_module = None +``` + +to + +```py +import numpy as array_module +``` + +(replacing `numpy` with the array module namespace to be tested). + +In either case, the tests should be run with the `pytest` command. + +Aside from the two testing dependencies (`pytest` and `hypothesis`), the test +suite has no dependencies. In particular, it does not depend on any specific +array libraries such as NumPy. All tests are run using only the array library +that is being tested, comparing results against the behavior as defined in the +spec. The test suite is designed to be standalone so that it can easily be vendored. -TODO: explain how to use the test suite to measure and report on the level of -(non-)conformance. +See the +[README](https://github.com/data-apis/array-api-tests/blob/master/README.md) +in the test suite repo for more information about how to run and interpret the +test suite results. From b51a56a5030965f2d95bf65b3e7eef6f27d174f1 Mon Sep 17 00:00:00 2001 From: Aaron Meurer Date: Mon, 26 Oct 2020 15:57:14 -0600 Subject: [PATCH 2/3] Fix Markdown link syntax --- spec/verification_test_suite.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/verification_test_suite.md b/spec/verification_test_suite.md index d00963a5d..04ad5422c 100644 --- a/spec/verification_test_suite.md +++ b/spec/verification_test_suite.md @@ -5,7 +5,7 @@ In addition to the specification documents, a test suite is being developed to aid library developers check conformance to the spec. **NOTE: The test suite is still a work in progress.** It can be found at -[[https://github.com/data-apis/array-api-tests]]. +. It is important to note that while the aim of the array api test suite is to cover as much as the spec as possible, there are necessarily some aspects of From 53bd7655f430b2e4ef70c167154a401d863498eb Mon Sep 17 00:00:00 2001 From: Ralf Gommers Date: Tue, 27 Oct 2020 15:52:31 +0000 Subject: [PATCH 3/3] Fix a couple of minor textual issues --- spec/verification_test_suite.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/spec/verification_test_suite.md b/spec/verification_test_suite.md index 04ad5422c..cbe770e48 100644 --- a/spec/verification_test_suite.md +++ b/spec/verification_test_suite.md @@ -7,8 +7,8 @@ aid library developers check conformance to the spec. **NOTE: The test suite is still a work in progress.** It can be found at . -It is important to note that while the aim of the array api test suite is to -cover as much as the spec as possible, there are necessarily some aspects of +It is important to note that while the aim of the array API test suite is to +cover as much of the spec as possible, there are necessarily some aspects of the spec that are not covered by the test suite, typically because they are impossible to effectively test. Furthermore, if the test suite appears to diverge in any way from what the spec documents say, this should be considered @@ -33,7 +33,7 @@ For example ARRAY_API_TESTS_MODULE=numpy pytest -Alternately, edit the `array_api_tests/_array_module.py` file and change the +Alternatively, edit the `array_api_tests/_array_module.py` file and change the line ```py