From fad0c39a9435daed17cb9e58320a96f54d3099cd Mon Sep 17 00:00:00 2001 From: Matthew Date: Wed, 22 Dec 2021 09:50:14 +0000 Subject: [PATCH 1/4] Streamlined content in coverage guide --- tests-coverage.md | 95 ++++++++--------------------------------------- 1 file changed, 15 insertions(+), 80 deletions(-) diff --git a/tests-coverage.md b/tests-coverage.md index 558b2238..9fb4e6d3 100644 --- a/tests-coverage.md +++ b/tests-coverage.md @@ -1,95 +1,30 @@ -## Coverage of the Test Suite +## What this suite actually tests -This document outlines the coverage of the test suite over the -[spec](https://data-apis.org/array-api/) at a high level. +`array-api-tests` tests that an array library adopting the [standard](https://data-apis.org/array-api/) is indeed covering everything that is in scope. -The following things are tested +## Primary tests -* **Smoke tested** means that the function has a basic test that calls the - function with some inputs, but does not imply any testing of the output - value. This includes calling keyword arguments to the function, and checking - that it takes the correct number of positional arguments. A smoke test will - fail if the function is not implemented with the correct signature or raises - an exception, but will not check any other aspect of the spec. +Every function—including array object methods—has a respective test method. We use [Hypothesis](https://hypothesis.readthedocs.io/en/latest/) to generate a diverse set of valid inputs. This means array inputs will cover different dtypes and shapes, as well as contain interesting elements. These examples generate with interesting arrangements of non-array positional arguments and keyword arguments. -* **All Inputs** means that the function is tested with all possible inputs - required by the spec (using hypothesis). This means all possible array - shapes, all possible dtypes (that are required for the given function), and - all possible values for the given dtype (omitting those whose behavior is - undefined). +Each test case will cover the following areas if relevant: -* **Output Shape** means that the result shape is tested. For functions that - take more than one argument, this means the result shape should produced - from - [broadcasting](https://data-apis.org/array-api/latest/API_specification/broadcasting.html) - the input shapes. For functions of a single argument, the result shape - should be the same as the input shape. +* **Smoking**: We pass our generated examples to all functions. As these examples solely consist of *valid* inputs, we are testing that functions can be called using their documented inputs without raising errors. -* **Output Dtype** means that the result dtype is tested. For (most) functions - with a single argument, the result dtype should be the same as the input. - For functions with two arguments, there are different possibilities, such as - performing [type - promotion](https://data-apis.org/array-api/latest/API_specification/type_promotion.html) - or always returning a specific dtype (e.g., `equals()` should always return - a `bool` array). +* **Data type**: For functions returning/modifying arrays, we assert that output arrays have the correct data types. Most functions [type-promote](https://data-apis.org/array-api/latest/API_specification/type_promotion.html) input arrays and some functions have bespoke rules—in both cases we simulate the correct behaviour to find the expected data types. -* **Output Values** means that the exact output is tested in some way. For - functions that operate on floating-point inputs, the spec does not require - exact values, so a "Yes" in this case will mean only that the output value - is checked to be "close" to the numerically correct result. The exception to - this is special cases for elementwise functions, which are tested exactly. - For functions that operate on non-floating-point inputs, or functions like - manipulation functions or indexing that simply rearrange the same values of - the input arrays, a "Yes" means that the exact values are tested. Note that - in many cases, certain values of inputs are left unspecified, and are thus - not tested (e.g., the behavior for division by integer 0 is unspecified). +* **Shape**: For functions returning/modifying arrays, we assert that output arrays have the correct shape. Most functions [broadcast](https://data-apis.org/array-api/latest/API_specification/broadcasting.html) input arrays and some functions have bespoke rules—in both cases we simulate the correct behaviour to find the expected shapes. -* **Stacking** means that functions that operate on "stacks" of smaller data - are tested to produce the same result on a stack as on the individual - components. For example, an elementwise function on an array - should produce the same output values as the same function called on each - value individually, or a linalg function on a stack of matrices should - produce the same value when called on individual matrices. Here "same" may - only mean "close" when the input values are floating-point. +* **Values**: We assert output values (including the elements of returned/modified arrays) are as expected. Except for manipulation functions or special cases, the spec allows floating-point inputs to have inexact outputs, so with such examples we only assert values are roughly as expected. -## Statistical Functions +## Additional tests -| Function | Smoke Test | All Inputs | Output Shape | Result Dtype | Output Values | Stacking | -|----------|------------|------------|--------------|--------------|---------------|----------| -| max | Yes | Yes | Yes | Yes | | | -| mean | Yes | Yes | Yes | Yes | | | -| min | Yes | Yes | Yes | Yes | | | -| prod | Yes | Yes | Yes | Yes [^1] | | | -| std | Yes | Yes | Yes | Yes | | | -| sum | Yes | Yes | Yes | Yes [^1] | | | -| var | Yes | Yes | Yes | Yes | | | +In addition to having one test case for each function, we test other properties of the functions and some miscellaneous things. -[^1]: `sum` and `prod` have special type promotion rules. +* **Special cases**: For functions with special case behaviour, we assert that these functions return the correct values. -## Additional Planned Features +* **Signatures**: We assert functions have the correct signatures. -In addition to getting full coverage of the spec, there are some additional -features and improvements for the test suite that are planned. Work on these features -will be guided primarily by concrete needs from library implementers, so if -you are someone using this test suite to test your library, please [let us -know](https://github.com/data-apis/array-api-tests/issues) the limitations you -come across. +* **Constants**: We assert that [constants](https://data-apis.org/array-api/latest/API_specification/constants.html) behave expectedly, are roughly the expected value, and that any related functions interact with them correctly. -- Making the test suite more usable for partially conforming libraries. Many - tests rely on various functions in the array library to function. This means - that if certain functions aren't implemented, for example, `asarray()` or - `equals()`, then many tests will not function at all. We want to improve - this situation, so that tests that don't strictly require these functions can - still be run. -- Better reporting. The pytest output can be difficult to parse, especially - when there are many failures. Additionally some error messages can be - difficult to understand without prior knowledge of the test internals. - Better reporting can also make it easier to compare different - implementations by their conformance. - -- Better tests for numerical outputs. Right now numerical outputs are either - not tested at all, or only tested against very rough epsilons. This is - partly due to the fact that the spec does not mandate any level of precision - for most functions. However, it may be useful to, for instance, give a - report of how off a given function is from the "expected" exact output. +TODO: future plans From 166dbf3b15d14da2a819278e712853976245afa2 Mon Sep 17 00:00:00 2001 From: Matthew Date: Wed, 22 Dec 2021 10:19:39 +0000 Subject: [PATCH 2/4] Rename `array_api_tests/` to `xptests/` --- .github/workflows/numpy.yml | 22 +++++++++---------- conftest.py | 6 ++--- generate_stubs.py | 16 +++++++------- {array_api_tests => xptests}/__init__.py | 0 {array_api_tests => xptests}/_array_module.py | 0 {array_api_tests => xptests}/algos.py | 0 {array_api_tests => xptests}/array_helpers.py | 0 {array_api_tests => xptests}/dtype_helpers.py | 0 .../function_stubs/__init__.py | 0 .../function_stubs/_types.py | 0 .../function_stubs/array_object.py | 0 .../function_stubs/constants.py | 0 .../function_stubs/creation_functions.py | 0 .../function_stubs/data_type_functions.py | 0 .../function_stubs/elementwise_functions.py | 0 .../function_stubs/linalg.py | 0 .../linear_algebra_functions.py | 0 .../function_stubs/manipulation_functions.py | 0 .../function_stubs/searching_functions.py | 0 .../function_stubs/set_functions.py | 0 .../function_stubs/sorting_functions.py | 0 .../function_stubs/statistical_functions.py | 0 .../function_stubs/utility_functions.py | 0 .../hypothesis_helpers.py | 0 {array_api_tests => xptests}/meta/__init__.py | 0 .../meta/test_array_helpers.py | 0 .../meta/test_broadcasting.py | 0 .../meta/test_hypothesis_helpers.py | 0 .../meta/test_partial_adopters.py | 0 .../meta/test_pytest_helpers.py | 0 .../meta/test_utils.py | 0 .../pytest_helpers.py | 0 {array_api_tests => xptests}/shape_helpers.py | 0 .../special_cases/__init__.py | 0 .../special_cases/test_abs.py | 0 .../special_cases/test_acos.py | 0 .../special_cases/test_acosh.py | 0 .../special_cases/test_add.py | 0 .../special_cases/test_asin.py | 0 .../special_cases/test_asinh.py | 0 .../special_cases/test_atan.py | 0 .../special_cases/test_atan2.py | 0 .../special_cases/test_atanh.py | 0 .../special_cases/test_ceil.py | 0 .../special_cases/test_cos.py | 0 .../special_cases/test_cosh.py | 0 .../special_cases/test_divide.py | 0 .../special_cases/test_dunder_abs.py | 0 .../special_cases/test_dunder_add.py | 0 .../special_cases/test_dunder_iadd.py | 0 .../special_cases/test_dunder_imul.py | 0 .../special_cases/test_dunder_ipow.py | 0 .../special_cases/test_dunder_itruediv.py | 0 .../special_cases/test_dunder_mul.py | 0 .../special_cases/test_dunder_pow.py | 0 .../special_cases/test_dunder_truediv.py | 0 .../special_cases/test_exp.py | 0 .../special_cases/test_expm1.py | 0 .../special_cases/test_floor.py | 0 .../special_cases/test_log.py | 0 .../special_cases/test_log10.py | 0 .../special_cases/test_log1p.py | 0 .../special_cases/test_log2.py | 0 .../special_cases/test_logaddexp.py | 0 .../special_cases/test_multiply.py | 0 .../special_cases/test_pow.py | 0 .../special_cases/test_round.py | 0 .../special_cases/test_sign.py | 0 .../special_cases/test_sin.py | 0 .../special_cases/test_sinh.py | 0 .../special_cases/test_sqrt.py | 0 .../special_cases/test_tan.py | 0 .../special_cases/test_tanh.py | 0 .../special_cases/test_trunc.py | 0 .../test_array2scalar.py | 0 .../test_constants.py | 0 .../test_creation_functions.py | 0 .../test_elementwise_functions.py | 0 {array_api_tests => xptests}/test_indexing.py | 0 {array_api_tests => xptests}/test_linalg.py | 0 .../test_manipulation_functions.py | 0 .../test_searching_functions.py | 0 .../test_set_functions.py | 0 .../test_signatures.py | 0 .../test_sorting_functions.py | 0 .../test_statistical_functions.py | 0 .../test_type_promotion.py | 0 .../test_utility_functions.py | 0 {array_api_tests => xptests}/typing.py | 0 89 files changed, 22 insertions(+), 22 deletions(-) rename {array_api_tests => xptests}/__init__.py (100%) rename {array_api_tests => xptests}/_array_module.py (100%) rename {array_api_tests => xptests}/algos.py (100%) rename {array_api_tests => xptests}/array_helpers.py (100%) rename {array_api_tests => xptests}/dtype_helpers.py (100%) rename {array_api_tests => xptests}/function_stubs/__init__.py (100%) rename {array_api_tests => xptests}/function_stubs/_types.py (100%) rename {array_api_tests => xptests}/function_stubs/array_object.py (100%) rename {array_api_tests => xptests}/function_stubs/constants.py (100%) rename {array_api_tests => xptests}/function_stubs/creation_functions.py (100%) rename {array_api_tests => xptests}/function_stubs/data_type_functions.py (100%) rename {array_api_tests => xptests}/function_stubs/elementwise_functions.py (100%) rename {array_api_tests => xptests}/function_stubs/linalg.py (100%) rename {array_api_tests => xptests}/function_stubs/linear_algebra_functions.py (100%) rename {array_api_tests => xptests}/function_stubs/manipulation_functions.py (100%) rename {array_api_tests => xptests}/function_stubs/searching_functions.py (100%) rename {array_api_tests => xptests}/function_stubs/set_functions.py (100%) rename {array_api_tests => xptests}/function_stubs/sorting_functions.py (100%) rename {array_api_tests => xptests}/function_stubs/statistical_functions.py (100%) rename {array_api_tests => xptests}/function_stubs/utility_functions.py (100%) rename {array_api_tests => xptests}/hypothesis_helpers.py (100%) rename {array_api_tests => xptests}/meta/__init__.py (100%) rename {array_api_tests => xptests}/meta/test_array_helpers.py (100%) rename {array_api_tests => xptests}/meta/test_broadcasting.py (100%) rename {array_api_tests => xptests}/meta/test_hypothesis_helpers.py (100%) rename {array_api_tests => xptests}/meta/test_partial_adopters.py (100%) rename {array_api_tests => xptests}/meta/test_pytest_helpers.py (100%) rename {array_api_tests => xptests}/meta/test_utils.py (100%) rename {array_api_tests => xptests}/pytest_helpers.py (100%) rename {array_api_tests => xptests}/shape_helpers.py (100%) rename {array_api_tests => xptests}/special_cases/__init__.py (100%) rename {array_api_tests => xptests}/special_cases/test_abs.py (100%) rename {array_api_tests => xptests}/special_cases/test_acos.py (100%) rename {array_api_tests => xptests}/special_cases/test_acosh.py (100%) rename {array_api_tests => xptests}/special_cases/test_add.py (100%) rename {array_api_tests => xptests}/special_cases/test_asin.py (100%) rename {array_api_tests => xptests}/special_cases/test_asinh.py (100%) rename {array_api_tests => xptests}/special_cases/test_atan.py (100%) rename {array_api_tests => xptests}/special_cases/test_atan2.py (100%) rename {array_api_tests => xptests}/special_cases/test_atanh.py (100%) rename {array_api_tests => xptests}/special_cases/test_ceil.py (100%) rename {array_api_tests => xptests}/special_cases/test_cos.py (100%) rename {array_api_tests => xptests}/special_cases/test_cosh.py (100%) rename {array_api_tests => xptests}/special_cases/test_divide.py (100%) rename {array_api_tests => xptests}/special_cases/test_dunder_abs.py (100%) rename {array_api_tests => xptests}/special_cases/test_dunder_add.py (100%) rename {array_api_tests => xptests}/special_cases/test_dunder_iadd.py (100%) rename {array_api_tests => xptests}/special_cases/test_dunder_imul.py (100%) rename {array_api_tests => xptests}/special_cases/test_dunder_ipow.py (100%) rename {array_api_tests => xptests}/special_cases/test_dunder_itruediv.py (100%) rename {array_api_tests => xptests}/special_cases/test_dunder_mul.py (100%) rename {array_api_tests => xptests}/special_cases/test_dunder_pow.py (100%) rename {array_api_tests => xptests}/special_cases/test_dunder_truediv.py (100%) rename {array_api_tests => xptests}/special_cases/test_exp.py (100%) rename {array_api_tests => xptests}/special_cases/test_expm1.py (100%) rename {array_api_tests => xptests}/special_cases/test_floor.py (100%) rename {array_api_tests => xptests}/special_cases/test_log.py (100%) rename {array_api_tests => xptests}/special_cases/test_log10.py (100%) rename {array_api_tests => xptests}/special_cases/test_log1p.py (100%) rename {array_api_tests => xptests}/special_cases/test_log2.py (100%) rename {array_api_tests => xptests}/special_cases/test_logaddexp.py (100%) rename {array_api_tests => xptests}/special_cases/test_multiply.py (100%) rename {array_api_tests => xptests}/special_cases/test_pow.py (100%) rename {array_api_tests => xptests}/special_cases/test_round.py (100%) rename {array_api_tests => xptests}/special_cases/test_sign.py (100%) rename {array_api_tests => xptests}/special_cases/test_sin.py (100%) rename {array_api_tests => xptests}/special_cases/test_sinh.py (100%) rename {array_api_tests => xptests}/special_cases/test_sqrt.py (100%) rename {array_api_tests => xptests}/special_cases/test_tan.py (100%) rename {array_api_tests => xptests}/special_cases/test_tanh.py (100%) rename {array_api_tests => xptests}/special_cases/test_trunc.py (100%) rename {array_api_tests => xptests}/test_array2scalar.py (100%) rename {array_api_tests => xptests}/test_constants.py (100%) rename {array_api_tests => xptests}/test_creation_functions.py (100%) rename {array_api_tests => xptests}/test_elementwise_functions.py (100%) rename {array_api_tests => xptests}/test_indexing.py (100%) rename {array_api_tests => xptests}/test_linalg.py (100%) rename {array_api_tests => xptests}/test_manipulation_functions.py (100%) rename {array_api_tests => xptests}/test_searching_functions.py (100%) rename {array_api_tests => xptests}/test_set_functions.py (100%) rename {array_api_tests => xptests}/test_signatures.py (100%) rename {array_api_tests => xptests}/test_sorting_functions.py (100%) rename {array_api_tests => xptests}/test_statistical_functions.py (100%) rename {array_api_tests => xptests}/test_type_promotion.py (100%) rename {array_api_tests => xptests}/test_utility_functions.py (100%) rename {array_api_tests => xptests}/typing.py (100%) diff --git a/.github/workflows/numpy.yml b/.github/workflows/numpy.yml index 262874e7..f8204952 100644 --- a/.github/workflows/numpy.yml +++ b/.github/workflows/numpy.yml @@ -29,24 +29,24 @@ jobs: cat << EOF >> xfails.txt # https://github.com/numpy/numpy/issues/18881 - array_api_tests/test_creation_functions.py::test_linspace + xptests/test_creation_functions.py::test_linspace # einsum is not yet completed in the spec - array_api_tests/test_signatures.py::test_has_names[einsum] + xptests/test_signatures.py::test_has_names[einsum] # dlpack support is not yet implemented in NumPy # See https://github.com/numpy/numpy/pull/19083 - array_api_tests/test_signatures.py::test_function_positional_args[__dlpack__] - array_api_tests/test_signatures.py::test_function_positional_args[__dlpack_device__] - array_api_tests/test_signatures.py::test_function_positional_args[from_dlpack] - array_api_tests/test_signatures.py::test_function_positional_args[to_device] - array_api_tests/test_signatures.py::test_function_keyword_only_args[__dlpack__] + xptests/test_signatures.py::test_function_positional_args[__dlpack__] + xptests/test_signatures.py::test_function_positional_args[__dlpack_device__] + xptests/test_signatures.py::test_function_positional_args[from_dlpack] + xptests/test_signatures.py::test_function_positional_args[to_device] + xptests/test_signatures.py::test_function_keyword_only_args[__dlpack__] # floor_divide has an issue related to https://github.com/data-apis/array-api/issues/264 - array_api_tests/test_elementwise_functions.py::test_floor_divide + xptests/test_elementwise_functions.py::test_floor_divide # mesgrid doesn't return all arrays as the promoted dtype - array_api_tests/test_type_promotion.py::test_meshgrid + xptests/test_type_promotion.py::test_meshgrid # https://github.com/numpy/numpy/pull/20066#issuecomment-947056094 - array_api_tests/test_type_promotion.py::test_where + xptests/test_type_promotion.py::test_where # shape mismatches are not handled - array_api_tests/test_type_promotion.py::test_tensordot + xptests/test_type_promotion.py::test_tensordot EOF diff --git a/conftest.py b/conftest.py index 64c46df7..cf4a5028 100644 --- a/conftest.py +++ b/conftest.py @@ -4,8 +4,8 @@ from pytest import mark from hypothesis import settings -from array_api_tests import _array_module as xp -from array_api_tests._array_module import _UndefinedStub +from xptests import _array_module as xp +from xptests._array_module import _UndefinedStub settings.register_profile('xp_default', deadline=800) @@ -77,7 +77,7 @@ def xp_has_ext(ext: str) -> bool: if xfails_path.exists(): with open(xfails_path) as f: for line in f: - if line.startswith('array_api_tests'): + if line.startswith('xptests'): id_ = line.strip('\n') xfail_ids.append(id_) diff --git a/generate_stubs.py b/generate_stubs.py index 3dcde542..ed2e8a43 100755 --- a/generate_stubs.py +++ b/generate_stubs.py @@ -7,7 +7,7 @@ ./generate_stubs.py path/to/clone/of/array-api -This will update the stub files in array_api_tests/function_stubs/ +This will update the stub files in xptests/function_stubs/ """ from __future__ import annotations @@ -218,12 +218,12 @@ def main(): parser.add_argument('-v', '--verbose', help="Print verbose output to the terminal", action='store_true') args = parser.parse_args() - types_path = os.path.join('array_api_tests', 'function_stubs', '_types.py') + types_path = os.path.join('xptests', 'function_stubs', '_types.py') if args.write: with open(types_path, 'w') as f: f.write(TYPES_HEADER) - special_cases_dir = Path('array_api_tests/special_cases') + special_cases_dir = Path('xptests/special_cases') special_cases_dir.mkdir(exist_ok=True) (special_cases_dir / '__init__.py').touch() @@ -256,7 +256,7 @@ def main(): title += " (Extension)" else: py_file = filename.replace('.md', '.py') - py_path = os.path.join('array_api_tests', 'function_stubs', py_file) + py_path = os.path.join('xptests', 'function_stubs', py_file) module_name = py_file.replace('.py', '') modules[module_name] = [] if args.verbose: @@ -342,7 +342,7 @@ def {annotated_sig}:{doc} if filename == 'elementwise_functions.md': special_cases = parse_special_cases(text, verbose=args.verbose) for func in special_cases: - py_path = os.path.join('array_api_tests', 'special_cases', f'test_{func}.py') + py_path = os.path.join('xptests', 'special_cases', f'test_{func}.py') tests = make_special_case_tests(func, special_cases, sigs) if tests: code = SPECIAL_CASES_HEADER.format(func=func) + '\n'.join(tests) @@ -354,7 +354,7 @@ def {annotated_sig}:{doc} elif filename == 'array_object.md': op_special_cases = parse_special_cases(text, verbose=args.verbose) for func in op_special_cases: - py_path = os.path.join('array_api_tests', 'special_cases', f'test_dunder_{func[2:-2]}.py') + py_path = os.path.join('xptests', 'special_cases', f'test_dunder_{func[2:-2]}.py') tests = make_special_case_tests(func, op_special_cases, sigs) if tests: code = OP_SPECIAL_CASES_HEADER.format(func=func) + '\n'.join(tests) @@ -368,7 +368,7 @@ def {annotated_sig}:{doc} iop = f"__i{name}__" iop_special_cases[iop] = op_special_cases[op] for func in iop_special_cases: - py_path = os.path.join('array_api_tests', 'special_cases', f'test_dunder_{func[2:-2]}.py') + py_path = os.path.join('xptests', 'special_cases', f'test_dunder_{func[2:-2]}.py') tests = make_special_case_tests(func, iop_special_cases, sigs) if tests: code = IOP_SPECIAL_CASES_HEADER.format(func=func, operator=func[2:-2]) + '\n'.join(tests) @@ -377,7 +377,7 @@ def {annotated_sig}:{doc} with open(py_path, 'w') as f: f.write(code) - init_path = os.path.join('array_api_tests', 'function_stubs', '__init__.py') + init_path = os.path.join('xptests', 'function_stubs', '__init__.py') if args.write: with open(init_path, 'w') as f: f.write(INIT_HEADER) diff --git a/array_api_tests/__init__.py b/xptests/__init__.py similarity index 100% rename from array_api_tests/__init__.py rename to xptests/__init__.py diff --git a/array_api_tests/_array_module.py b/xptests/_array_module.py similarity index 100% rename from array_api_tests/_array_module.py rename to xptests/_array_module.py diff --git a/array_api_tests/algos.py b/xptests/algos.py similarity index 100% rename from array_api_tests/algos.py rename to xptests/algos.py diff --git a/array_api_tests/array_helpers.py b/xptests/array_helpers.py similarity index 100% rename from array_api_tests/array_helpers.py rename to xptests/array_helpers.py diff --git a/array_api_tests/dtype_helpers.py b/xptests/dtype_helpers.py similarity index 100% rename from array_api_tests/dtype_helpers.py rename to xptests/dtype_helpers.py diff --git a/array_api_tests/function_stubs/__init__.py b/xptests/function_stubs/__init__.py similarity index 100% rename from array_api_tests/function_stubs/__init__.py rename to xptests/function_stubs/__init__.py diff --git a/array_api_tests/function_stubs/_types.py b/xptests/function_stubs/_types.py similarity index 100% rename from array_api_tests/function_stubs/_types.py rename to xptests/function_stubs/_types.py diff --git a/array_api_tests/function_stubs/array_object.py b/xptests/function_stubs/array_object.py similarity index 100% rename from array_api_tests/function_stubs/array_object.py rename to xptests/function_stubs/array_object.py diff --git a/array_api_tests/function_stubs/constants.py b/xptests/function_stubs/constants.py similarity index 100% rename from array_api_tests/function_stubs/constants.py rename to xptests/function_stubs/constants.py diff --git a/array_api_tests/function_stubs/creation_functions.py b/xptests/function_stubs/creation_functions.py similarity index 100% rename from array_api_tests/function_stubs/creation_functions.py rename to xptests/function_stubs/creation_functions.py diff --git a/array_api_tests/function_stubs/data_type_functions.py b/xptests/function_stubs/data_type_functions.py similarity index 100% rename from array_api_tests/function_stubs/data_type_functions.py rename to xptests/function_stubs/data_type_functions.py diff --git a/array_api_tests/function_stubs/elementwise_functions.py b/xptests/function_stubs/elementwise_functions.py similarity index 100% rename from array_api_tests/function_stubs/elementwise_functions.py rename to xptests/function_stubs/elementwise_functions.py diff --git a/array_api_tests/function_stubs/linalg.py b/xptests/function_stubs/linalg.py similarity index 100% rename from array_api_tests/function_stubs/linalg.py rename to xptests/function_stubs/linalg.py diff --git a/array_api_tests/function_stubs/linear_algebra_functions.py b/xptests/function_stubs/linear_algebra_functions.py similarity index 100% rename from array_api_tests/function_stubs/linear_algebra_functions.py rename to xptests/function_stubs/linear_algebra_functions.py diff --git a/array_api_tests/function_stubs/manipulation_functions.py b/xptests/function_stubs/manipulation_functions.py similarity index 100% rename from array_api_tests/function_stubs/manipulation_functions.py rename to xptests/function_stubs/manipulation_functions.py diff --git a/array_api_tests/function_stubs/searching_functions.py b/xptests/function_stubs/searching_functions.py similarity index 100% rename from array_api_tests/function_stubs/searching_functions.py rename to xptests/function_stubs/searching_functions.py diff --git a/array_api_tests/function_stubs/set_functions.py b/xptests/function_stubs/set_functions.py similarity index 100% rename from array_api_tests/function_stubs/set_functions.py rename to xptests/function_stubs/set_functions.py diff --git a/array_api_tests/function_stubs/sorting_functions.py b/xptests/function_stubs/sorting_functions.py similarity index 100% rename from array_api_tests/function_stubs/sorting_functions.py rename to xptests/function_stubs/sorting_functions.py diff --git a/array_api_tests/function_stubs/statistical_functions.py b/xptests/function_stubs/statistical_functions.py similarity index 100% rename from array_api_tests/function_stubs/statistical_functions.py rename to xptests/function_stubs/statistical_functions.py diff --git a/array_api_tests/function_stubs/utility_functions.py b/xptests/function_stubs/utility_functions.py similarity index 100% rename from array_api_tests/function_stubs/utility_functions.py rename to xptests/function_stubs/utility_functions.py diff --git a/array_api_tests/hypothesis_helpers.py b/xptests/hypothesis_helpers.py similarity index 100% rename from array_api_tests/hypothesis_helpers.py rename to xptests/hypothesis_helpers.py diff --git a/array_api_tests/meta/__init__.py b/xptests/meta/__init__.py similarity index 100% rename from array_api_tests/meta/__init__.py rename to xptests/meta/__init__.py diff --git a/array_api_tests/meta/test_array_helpers.py b/xptests/meta/test_array_helpers.py similarity index 100% rename from array_api_tests/meta/test_array_helpers.py rename to xptests/meta/test_array_helpers.py diff --git a/array_api_tests/meta/test_broadcasting.py b/xptests/meta/test_broadcasting.py similarity index 100% rename from array_api_tests/meta/test_broadcasting.py rename to xptests/meta/test_broadcasting.py diff --git a/array_api_tests/meta/test_hypothesis_helpers.py b/xptests/meta/test_hypothesis_helpers.py similarity index 100% rename from array_api_tests/meta/test_hypothesis_helpers.py rename to xptests/meta/test_hypothesis_helpers.py diff --git a/array_api_tests/meta/test_partial_adopters.py b/xptests/meta/test_partial_adopters.py similarity index 100% rename from array_api_tests/meta/test_partial_adopters.py rename to xptests/meta/test_partial_adopters.py diff --git a/array_api_tests/meta/test_pytest_helpers.py b/xptests/meta/test_pytest_helpers.py similarity index 100% rename from array_api_tests/meta/test_pytest_helpers.py rename to xptests/meta/test_pytest_helpers.py diff --git a/array_api_tests/meta/test_utils.py b/xptests/meta/test_utils.py similarity index 100% rename from array_api_tests/meta/test_utils.py rename to xptests/meta/test_utils.py diff --git a/array_api_tests/pytest_helpers.py b/xptests/pytest_helpers.py similarity index 100% rename from array_api_tests/pytest_helpers.py rename to xptests/pytest_helpers.py diff --git a/array_api_tests/shape_helpers.py b/xptests/shape_helpers.py similarity index 100% rename from array_api_tests/shape_helpers.py rename to xptests/shape_helpers.py diff --git a/array_api_tests/special_cases/__init__.py b/xptests/special_cases/__init__.py similarity index 100% rename from array_api_tests/special_cases/__init__.py rename to xptests/special_cases/__init__.py diff --git a/array_api_tests/special_cases/test_abs.py b/xptests/special_cases/test_abs.py similarity index 100% rename from array_api_tests/special_cases/test_abs.py rename to xptests/special_cases/test_abs.py diff --git a/array_api_tests/special_cases/test_acos.py b/xptests/special_cases/test_acos.py similarity index 100% rename from array_api_tests/special_cases/test_acos.py rename to xptests/special_cases/test_acos.py diff --git a/array_api_tests/special_cases/test_acosh.py b/xptests/special_cases/test_acosh.py similarity index 100% rename from array_api_tests/special_cases/test_acosh.py rename to xptests/special_cases/test_acosh.py diff --git a/array_api_tests/special_cases/test_add.py b/xptests/special_cases/test_add.py similarity index 100% rename from array_api_tests/special_cases/test_add.py rename to xptests/special_cases/test_add.py diff --git a/array_api_tests/special_cases/test_asin.py b/xptests/special_cases/test_asin.py similarity index 100% rename from array_api_tests/special_cases/test_asin.py rename to xptests/special_cases/test_asin.py diff --git a/array_api_tests/special_cases/test_asinh.py b/xptests/special_cases/test_asinh.py similarity index 100% rename from array_api_tests/special_cases/test_asinh.py rename to xptests/special_cases/test_asinh.py diff --git a/array_api_tests/special_cases/test_atan.py b/xptests/special_cases/test_atan.py similarity index 100% rename from array_api_tests/special_cases/test_atan.py rename to xptests/special_cases/test_atan.py diff --git a/array_api_tests/special_cases/test_atan2.py b/xptests/special_cases/test_atan2.py similarity index 100% rename from array_api_tests/special_cases/test_atan2.py rename to xptests/special_cases/test_atan2.py diff --git a/array_api_tests/special_cases/test_atanh.py b/xptests/special_cases/test_atanh.py similarity index 100% rename from array_api_tests/special_cases/test_atanh.py rename to xptests/special_cases/test_atanh.py diff --git a/array_api_tests/special_cases/test_ceil.py b/xptests/special_cases/test_ceil.py similarity index 100% rename from array_api_tests/special_cases/test_ceil.py rename to xptests/special_cases/test_ceil.py diff --git a/array_api_tests/special_cases/test_cos.py b/xptests/special_cases/test_cos.py similarity index 100% rename from array_api_tests/special_cases/test_cos.py rename to xptests/special_cases/test_cos.py diff --git a/array_api_tests/special_cases/test_cosh.py b/xptests/special_cases/test_cosh.py similarity index 100% rename from array_api_tests/special_cases/test_cosh.py rename to xptests/special_cases/test_cosh.py diff --git a/array_api_tests/special_cases/test_divide.py b/xptests/special_cases/test_divide.py similarity index 100% rename from array_api_tests/special_cases/test_divide.py rename to xptests/special_cases/test_divide.py diff --git a/array_api_tests/special_cases/test_dunder_abs.py b/xptests/special_cases/test_dunder_abs.py similarity index 100% rename from array_api_tests/special_cases/test_dunder_abs.py rename to xptests/special_cases/test_dunder_abs.py diff --git a/array_api_tests/special_cases/test_dunder_add.py b/xptests/special_cases/test_dunder_add.py similarity index 100% rename from array_api_tests/special_cases/test_dunder_add.py rename to xptests/special_cases/test_dunder_add.py diff --git a/array_api_tests/special_cases/test_dunder_iadd.py b/xptests/special_cases/test_dunder_iadd.py similarity index 100% rename from array_api_tests/special_cases/test_dunder_iadd.py rename to xptests/special_cases/test_dunder_iadd.py diff --git a/array_api_tests/special_cases/test_dunder_imul.py b/xptests/special_cases/test_dunder_imul.py similarity index 100% rename from array_api_tests/special_cases/test_dunder_imul.py rename to xptests/special_cases/test_dunder_imul.py diff --git a/array_api_tests/special_cases/test_dunder_ipow.py b/xptests/special_cases/test_dunder_ipow.py similarity index 100% rename from array_api_tests/special_cases/test_dunder_ipow.py rename to xptests/special_cases/test_dunder_ipow.py diff --git a/array_api_tests/special_cases/test_dunder_itruediv.py b/xptests/special_cases/test_dunder_itruediv.py similarity index 100% rename from array_api_tests/special_cases/test_dunder_itruediv.py rename to xptests/special_cases/test_dunder_itruediv.py diff --git a/array_api_tests/special_cases/test_dunder_mul.py b/xptests/special_cases/test_dunder_mul.py similarity index 100% rename from array_api_tests/special_cases/test_dunder_mul.py rename to xptests/special_cases/test_dunder_mul.py diff --git a/array_api_tests/special_cases/test_dunder_pow.py b/xptests/special_cases/test_dunder_pow.py similarity index 100% rename from array_api_tests/special_cases/test_dunder_pow.py rename to xptests/special_cases/test_dunder_pow.py diff --git a/array_api_tests/special_cases/test_dunder_truediv.py b/xptests/special_cases/test_dunder_truediv.py similarity index 100% rename from array_api_tests/special_cases/test_dunder_truediv.py rename to xptests/special_cases/test_dunder_truediv.py diff --git a/array_api_tests/special_cases/test_exp.py b/xptests/special_cases/test_exp.py similarity index 100% rename from array_api_tests/special_cases/test_exp.py rename to xptests/special_cases/test_exp.py diff --git a/array_api_tests/special_cases/test_expm1.py b/xptests/special_cases/test_expm1.py similarity index 100% rename from array_api_tests/special_cases/test_expm1.py rename to xptests/special_cases/test_expm1.py diff --git a/array_api_tests/special_cases/test_floor.py b/xptests/special_cases/test_floor.py similarity index 100% rename from array_api_tests/special_cases/test_floor.py rename to xptests/special_cases/test_floor.py diff --git a/array_api_tests/special_cases/test_log.py b/xptests/special_cases/test_log.py similarity index 100% rename from array_api_tests/special_cases/test_log.py rename to xptests/special_cases/test_log.py diff --git a/array_api_tests/special_cases/test_log10.py b/xptests/special_cases/test_log10.py similarity index 100% rename from array_api_tests/special_cases/test_log10.py rename to xptests/special_cases/test_log10.py diff --git a/array_api_tests/special_cases/test_log1p.py b/xptests/special_cases/test_log1p.py similarity index 100% rename from array_api_tests/special_cases/test_log1p.py rename to xptests/special_cases/test_log1p.py diff --git a/array_api_tests/special_cases/test_log2.py b/xptests/special_cases/test_log2.py similarity index 100% rename from array_api_tests/special_cases/test_log2.py rename to xptests/special_cases/test_log2.py diff --git a/array_api_tests/special_cases/test_logaddexp.py b/xptests/special_cases/test_logaddexp.py similarity index 100% rename from array_api_tests/special_cases/test_logaddexp.py rename to xptests/special_cases/test_logaddexp.py diff --git a/array_api_tests/special_cases/test_multiply.py b/xptests/special_cases/test_multiply.py similarity index 100% rename from array_api_tests/special_cases/test_multiply.py rename to xptests/special_cases/test_multiply.py diff --git a/array_api_tests/special_cases/test_pow.py b/xptests/special_cases/test_pow.py similarity index 100% rename from array_api_tests/special_cases/test_pow.py rename to xptests/special_cases/test_pow.py diff --git a/array_api_tests/special_cases/test_round.py b/xptests/special_cases/test_round.py similarity index 100% rename from array_api_tests/special_cases/test_round.py rename to xptests/special_cases/test_round.py diff --git a/array_api_tests/special_cases/test_sign.py b/xptests/special_cases/test_sign.py similarity index 100% rename from array_api_tests/special_cases/test_sign.py rename to xptests/special_cases/test_sign.py diff --git a/array_api_tests/special_cases/test_sin.py b/xptests/special_cases/test_sin.py similarity index 100% rename from array_api_tests/special_cases/test_sin.py rename to xptests/special_cases/test_sin.py diff --git a/array_api_tests/special_cases/test_sinh.py b/xptests/special_cases/test_sinh.py similarity index 100% rename from array_api_tests/special_cases/test_sinh.py rename to xptests/special_cases/test_sinh.py diff --git a/array_api_tests/special_cases/test_sqrt.py b/xptests/special_cases/test_sqrt.py similarity index 100% rename from array_api_tests/special_cases/test_sqrt.py rename to xptests/special_cases/test_sqrt.py diff --git a/array_api_tests/special_cases/test_tan.py b/xptests/special_cases/test_tan.py similarity index 100% rename from array_api_tests/special_cases/test_tan.py rename to xptests/special_cases/test_tan.py diff --git a/array_api_tests/special_cases/test_tanh.py b/xptests/special_cases/test_tanh.py similarity index 100% rename from array_api_tests/special_cases/test_tanh.py rename to xptests/special_cases/test_tanh.py diff --git a/array_api_tests/special_cases/test_trunc.py b/xptests/special_cases/test_trunc.py similarity index 100% rename from array_api_tests/special_cases/test_trunc.py rename to xptests/special_cases/test_trunc.py diff --git a/array_api_tests/test_array2scalar.py b/xptests/test_array2scalar.py similarity index 100% rename from array_api_tests/test_array2scalar.py rename to xptests/test_array2scalar.py diff --git a/array_api_tests/test_constants.py b/xptests/test_constants.py similarity index 100% rename from array_api_tests/test_constants.py rename to xptests/test_constants.py diff --git a/array_api_tests/test_creation_functions.py b/xptests/test_creation_functions.py similarity index 100% rename from array_api_tests/test_creation_functions.py rename to xptests/test_creation_functions.py diff --git a/array_api_tests/test_elementwise_functions.py b/xptests/test_elementwise_functions.py similarity index 100% rename from array_api_tests/test_elementwise_functions.py rename to xptests/test_elementwise_functions.py diff --git a/array_api_tests/test_indexing.py b/xptests/test_indexing.py similarity index 100% rename from array_api_tests/test_indexing.py rename to xptests/test_indexing.py diff --git a/array_api_tests/test_linalg.py b/xptests/test_linalg.py similarity index 100% rename from array_api_tests/test_linalg.py rename to xptests/test_linalg.py diff --git a/array_api_tests/test_manipulation_functions.py b/xptests/test_manipulation_functions.py similarity index 100% rename from array_api_tests/test_manipulation_functions.py rename to xptests/test_manipulation_functions.py diff --git a/array_api_tests/test_searching_functions.py b/xptests/test_searching_functions.py similarity index 100% rename from array_api_tests/test_searching_functions.py rename to xptests/test_searching_functions.py diff --git a/array_api_tests/test_set_functions.py b/xptests/test_set_functions.py similarity index 100% rename from array_api_tests/test_set_functions.py rename to xptests/test_set_functions.py diff --git a/array_api_tests/test_signatures.py b/xptests/test_signatures.py similarity index 100% rename from array_api_tests/test_signatures.py rename to xptests/test_signatures.py diff --git a/array_api_tests/test_sorting_functions.py b/xptests/test_sorting_functions.py similarity index 100% rename from array_api_tests/test_sorting_functions.py rename to xptests/test_sorting_functions.py diff --git a/array_api_tests/test_statistical_functions.py b/xptests/test_statistical_functions.py similarity index 100% rename from array_api_tests/test_statistical_functions.py rename to xptests/test_statistical_functions.py diff --git a/array_api_tests/test_type_promotion.py b/xptests/test_type_promotion.py similarity index 100% rename from array_api_tests/test_type_promotion.py rename to xptests/test_type_promotion.py diff --git a/array_api_tests/test_utility_functions.py b/xptests/test_utility_functions.py similarity index 100% rename from array_api_tests/test_utility_functions.py rename to xptests/test_utility_functions.py diff --git a/array_api_tests/typing.py b/xptests/typing.py similarity index 100% rename from array_api_tests/typing.py rename to xptests/typing.py From 41dc5c83eca7d577cd300ac18174d77856acea73 Mon Sep 17 00:00:00 2001 From: Matthew Date: Wed, 22 Dec 2021 10:26:56 +0000 Subject: [PATCH 3/4] Rename `ARRAY_API_TESTS_MODULE` env to `XPTESTS` --- .github/workflows/numpy.yml | 2 +- xptests/_array_module.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/numpy.yml b/.github/workflows/numpy.yml index f8204952..c8ab1692 100644 --- a/.github/workflows/numpy.yml +++ b/.github/workflows/numpy.yml @@ -23,7 +23,7 @@ jobs: python -m pip install -r requirements.txt - name: Run the test suite env: - ARRAY_API_TESTS_MODULE: numpy.array_api + XPTESTS_MODULE: numpy.array_api run: | # Mark some known issues as XFAIL cat << EOF >> xfails.txt diff --git a/xptests/_array_module.py b/xptests/_array_module.py index 3ec4e3c4..15c50271 100644 --- a/xptests/_array_module.py +++ b/xptests/_array_module.py @@ -9,8 +9,8 @@ array_module = None if array_module is None: - if 'ARRAY_API_TESTS_MODULE' in os.environ: - mod_name = os.environ['ARRAY_API_TESTS_MODULE'] + if 'XPTESTS_MODULE' in os.environ: + mod_name = os.environ['XPTESTS_MODULE'] _module, _sub = mod_name, None if '.' in mod_name: _module, _sub = mod_name.split('.', 1) @@ -24,7 +24,7 @@ # submodules that can be imported (like mxnet.nd). mod = import_module(mod_name) else: - raise RuntimeError("No array module specified. Either edit _array_module.py or set the ARRAY_API_TESTS_MODULE environment variable") + raise RuntimeError("No array module specified. Either edit _array_module.py or set the XPTESTS_MODULE environment variable") else: mod = array_module mod_name = mod.__name__ From 5a4bc96689362a03e6376a3e069230602829bb45 Mon Sep 17 00:00:00 2001 From: Matthew Date: Wed, 22 Dec 2021 20:10:30 +0000 Subject: [PATCH 4/4] README is more succint and covers more topics --- README.md | 270 +++++++++++++++++++++++++++------------------- tests-coverage.md | 30 ------ 2 files changed, 158 insertions(+), 142 deletions(-) delete mode 100644 tests-coverage.md diff --git a/README.md b/README.md index c8d30717..5ceb1ad6 100644 --- a/README.md +++ b/README.md @@ -1,83 +1,137 @@ -# Array API Standard Test Suite +# Test Suite for Array API Compliance -This is the test suite for the Python array API standard. +This is the test suite for array libraries adopting the [Python Array API +standard](https://data-apis.org/array-api/). -**NOTE: This test suite is still a work in progress.** +Note the suite is still a **work in progress**. Feedback and contributions are +welcome! -Feedback and contributions are welcome, but be aware that this suite is not -yet completed. In particular, there are still many parts of the array API -specification that are not yet tested here. - -## Running the tests +## Quickstart ### Setup -To run the tests, first install the testing dependencies +To run the tests, install the testing dependencies. + +```bash +$ pip install -r requirements +``` - pip install pytest hypothesis +Ensure you have the array library that you want to test installed. -or +### Specifying the array module - conda install pytest hypothesis +You need to specify the array library to test. It can be specified via the +`XPTESTS_MODULE` environment variable, e.g. -as well as the array libraries that you want to test. +```bash +$ export XPTESTS_MODULE=numpy.array_api +``` -### Specifying the array module +Alternately, change the `array_module` variable in `xptests/_array_module.py` +line, e.g. -To run the tests, you need to set the array library that is to be tested. There -are two ways to do this. One way is to set the `ARRAY_API_TESTS_MODULE` -environment variable. For example you can set it when running `pytest` +```diff +- array_module = None ++ import numpy.array_api as array_module +``` - ARRAY_API_TESTS_MODULE=numpy pytest +### Run the suite -Alternately, edit the `array_api_tests/_array_module.py` file and change the -line +Simply run `pytest` against the `xptests/` folder to run the full suite. -```py -array_module = None +```bash +$ pytest xptests/ ``` -to +The suite tries to logically organise its tests. `pytest` allows you to only run +a specific test case, which is useful when developing functions. -```py -import numpy as array_module +```bash +$ pytest xptests/test_creation_functions.py::test_zeros ``` -(replacing `numpy` with the array module namespace to be tested). +## What the test suite covers + +We are interested in array libraries conforming to the +[spec](https://data-apis.org/array-api/latest/API_specification/index.html). +Ideally this means that if a library has fully adopted the Array API, the test +suite passes. We take great care to _not_ test things which are out-of-scope, so +as to not unexpectedly fail the suite. + +### Primary tests + +Every function—including array object methods—has a respective test method. We +use [Hypothesis](https://hypothesis.readthedocs.io/en/latest/) to generate a +diverse set of valid inputs. This means array inputs will cover different dtypes +and shapes, as well as contain interesting elements. These examples generate +with interesting arrangements of non-array positional arguments and keyword +arguments. + +Each test case will cover the following areas if relevant: + +* **Smoking**: We pass our generated examples to all functions. As these + examples solely consist of *valid* inputs, we are testing that functions can + be called using their documented inputs without raising errors. -### Specifying test cases +* **Data type**: For functions returning/modifying arrays, we assert that output + arrays have the correct data types. Most functions + [type-promote](https://data-apis.org/array-api/latest/API_specification/type_promotion.html) + input arrays and some functions have bespoke rules—in both cases we simulate + the correct behaviour to find the expected data types. -The test suite tries to logically organise its tests so you can find specific -test cases whilst developing something in particular. So to avoid running the -rather slow complete suite, you can specify particular test cases like any other -test suite. +* **Shape**: For functions returning/modifying arrays, we assert that output + arrays have the correct shape. Most functions + [broadcast](https://data-apis.org/array-api/latest/API_specification/broadcasting.html) + input arrays and some functions have bespoke rules—in both cases we simulate + the correct behaviour to find the expected shapes. - pytest array_api_tests/test_creation_functions.py::test_zeros +* **Values**: We assert output values (including the elements of + returned/modified arrays) are as expected. Except for manipulation functions + or special cases, the spec allows floating-point inputs to have inexact + outputs, so with such examples we only assert values are roughly as expected. -## Notes on Interpreting Errors +### Additional tests -- Some tests cannot be run unless other tests pass first. This is because very - basic APIs such as certain array creation APIs are required for a large - fraction of the tests to run. TODO: Write which tests are required to pass - first here. +In addition to having one test case for each function, we test other properties +of the functions and some miscellaneous things. -- If an error message involves `_UndefinedStub`, it means some name that is - required for the test to run is not defined in the array library. +* **Special cases**: For functions with special case behaviour, we assert that + these functions return the correct values. -- Due to the nature of the array api spec, virtually every array library will - produce a large number of errors from nonconformance. It is still a work in - progress to enable reporting the errors in a way that makes them easy to - understand, even if there are a large number of them. +* **Signatures**: We assert functions have the correct signatures. -- The spec documents are the ground source of truth. If the test suite appears - to be testing something that is different from the spec, or something that - isn't actually mentioned in the spec, this is a bug. [Please report - it](https://github.com/data-apis/array-api-tests/issues/new). Furthermore, - be aware that some aspects of the spec are either impossible or extremely - difficult to actually test, so they are not covered in the test suite (TODO: - list what these are). +* **Constants**: We assert that + [constants](https://data-apis.org/array-api/latest/API_specification/constants.html) + behave expectedly, are roughly the expected value, and that any related + functions interact with them correctly. + +Be aware that some aspects of the spec are impractical or impossible to actually +test, so they are not covered in the suite -## Configuring Tests +## Interpreting errors + +First and foremost, note that most tests have to assume that certain aspects of +the Array API have been correctly adopted, as fundamental APIs such as array +creation and equalities are hard requirements for many assertions. This means a +test case for one function might fail because another function has bugs or even +no implementation. + +This means adopting libraries at first will result in a vast number of errors +due to cascading errors. Generally the nature of the spec means many granular +details such as type promotion is likely going to also fail nearly-conforming +functions. + +We hope to improve user experience in regards to "noisy" errors in +[#51](https://github.com/data-apis/array-api-tests/issues/51). For now, if an +error message involves `_UndefinedStub`, it means an attribute of the array +library (including functions) and it's objects (e.g. the array) is missing. + +The spec is the suite's source of truth. If the suite appears to assume +behaviour different from the spec, or test something that is not documented, +this is a bug—please [report such +issues](https://github.com/data-apis/array-api-tests/issues/) to us. + +## Configuration By default, tests for the optional Array API extensions such as [`linalg`](https://data-apis.org/array-api/latest/extensions/linear_algebra_functions.html) @@ -85,82 +139,74 @@ will be skipped if not present in the specified array module. You can purposely skip testing extension(s) via the `--disable-extension` option, and likewise purposely test them via the `--enable-extension` option. -The tests make heavy use of the -[Hypothesis](https://hypothesis.readthedocs.io/en/latest/) testing library. -Hypothesis generates random input values for the tests. You can configure how -many values are generated and run using the `--max-examples` flag. The default -`--max-examples` is 100. For example, `--max-examples 50` will only generate -half as many examples and as a result, the test suite will run in about half -the time. Setting `--max-examples` to a lower value can be useful when you -want to have a faster test run. It can also be useful to set `--max-examples` -to a large value to do a longer, more rigorous run of the tests. For example, -`--max-examples 10000` will do a very rigorous check of the tests, but may -take a few hours to run. - -## Contributing - -### Adding Tests +The tests make heavy use +[Hypothesis](https://hypothesis.readthedocs.io/en/latest/). You can configure +how many examples are generated using the `--max-examples` flag, which defaults +to 100. Lower values can be useful for quick checks, and larger values should +result in more rigorous runs. For example, `--max-examples 10000` may find bugs +where default runs don't, but will take a much longer time. -It is important that every test in the test suite only uses APIs that are part -of the standard. This means that, for instance, when creating test arrays, you -should only use array creation functions that are part of the spec, such as -`ones` or `full`. It also means that many array testing functions that are -built-in to libraries like numpy are reimplemented in the test suite (see -`array_api_tests/pytest_helpers.py`, `array_api_tests/array_helpers.py`, and -`array_api_tests/hypothesis_helpers.py`). + -In order to enforce this, the `array_api_tests._array_module` should be used -everywhere in place of the actual array module that is being tested. +## Contributing -### Hypothesis +### Remain in-scope -The test suite uses [Hypothesis](https://hypothesis.readthedocs.io/en/latest/) -to generate random input data. Any test that should be applied over all -possible array inputs should use hypothesis tests. Custom Hypothesis -strategies are in the `array_api_tests/hypothesis_helpers.py` file. +It is important that every test only uses APIs that are part of the standard. +For instance, when creating input arrays you should only use the [array creation +functions](https://data-apis.org/array-api/latest/API_specification/creation_functions.html) +that are documented in the spec. The same goes for testing arrays—you'll find +many utilities that parralel NumPy's own test utils in the `*_helpers.py` files. -### Parameterization +### Tools -Any test that applies over all functions in a module should use -`pytest.mark.parametrize` to parameterize over them. For example, +Hypothesis should always be used for the primary tests, and can be useful +elsewhere. Effort should be made so drawn arguments are labeled with their +respective names. For +[`st.data()`](https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.strategies.data), +draws should be accompanied with the `label` kwarg i.e. `data.draw(, +label=