CI environments overhaul (#3340)

* Rationalize and align CI environments. Add many optional dependencies to individual CI suites.

* pynio and cdms2 are not available on Windows

* cfgrib causes Python interpreter crash on Windows

* dtype of np.arange defaults to int64 on Linux and int32 on Windows

* Suppress failure to delete file on Windows

* Mark hypotesis tests as @slow

Author: crusaderky

azure-pipelines.yml

@@ -17,7 +17,7 @@ jobs:
       py37-upstream-dev:
         conda_env: py37
         upstream_dev: true
-      py36-flakey:
+      py36-flaky:
         conda_env: py36
         pytest_extra_flags: --run-flaky --run-network-tests
         allow_failure: true
@@ -90,15 +90,3 @@ jobs:
       cd doc
       sphinx-build -n -j auto -b html -d _build/doctrees . _build/html
     displayName: Build HTML docs
-
-- job: LinuxHypothesis
-  variables:
-    conda_env: py36-hypothesis
-  pool:
-    vmImage: 'ubuntu-16.04'
-  steps:
-  - template: ci/azure/install.yml
-  - bash: |
-      source activate xarray-tests
-      pytest properties
-    displayName: Property based tests

ci/azure/unit-tests.yml

@@ -11,7 +11,7 @@ steps:
 # https://github.com/microsoft/azure-pipelines-tasks/issues/9302
 - bash: |
     source activate xarray-tests
-    pytest xarray \
+    pytest \
     --junitxml=junit/test-results.xml \
     --cov=xarray \
     --cov-report=xml \

ci/requirements/py36-hypothesis.yml

@@ -3,36 +3,40 @@ channels:
   - conda-forge
 dependencies:
   - python=3.6
+  - black
+  - boto3
+  - bottleneck
+  - cdms2
+  - cfgrib>=0.9.2
   - cftime
+  - coveralls
   - dask
   - distributed
-  - h5py
+  - flake8
   - h5netcdf
+  - h5py
+  - hypothesis
+  - iris>=1.10
+  - lxml
   - matplotlib
-  - netcdf4
-  - pytest
-  - pytest-cov
-  - pytest-env
-  - coveralls
-  - flake8
   - mypy==0.720  # Must match .pre-commit-config.yaml
+  - nc-time-axis
+  - netcdf4
+  - numba
   - numpy>=1.12
   - pandas>=0.19
   - pip
+  - pseudonetcdf>=3.0.1
+  - pydap
+  - pynio
+  - pytest
+  - pytest-cov
+  - pytest-env
+  - rasterio
   - scipy
   - seaborn
+  - sparse
   - toolz
-  - rasterio
-  - boto3
-  - bottleneck
   - zarr
-  - pseudonetcdf>=3.0.1
-  - cfgrib>=0.9.2
-  - cdms2
-  # Disabled due to conflicting dependencies:
-  # https://github.com/pydata/xarray/issues/3154
-  # - pynio
-  - iris>=1.10
-  - pydap
-  - lxml
-
+  - pip:
+    - numbagg

ci/requirements/py36.yml

@@ -3,24 +3,40 @@ channels:
   - conda-forge
 dependencies:
   - python=3.7
+  - black
+  - boto3
+  - bottleneck
+  # - cdms2  # Not available on Windows
+  # - cfgrib>=0.9.2  # Causes Python interpreter crash on Windows
   - cftime
-  - nc-time-axis
+  - coveralls
   - dask
   - distributed
-  - h5py
+  - flake8
   - h5netcdf
+  - h5py
+  - hypothesis
+  - iris>=1.10
+  - lxml
   - matplotlib
+  - mypy==0.720  # Must match .pre-commit-config.yaml
+  - nc-time-axis
   - netcdf4
-  - pytest
-  - pytest-cov
-  - pytest-env
-  - coveralls
+  - numba
   - numpy>=1.12
   - pandas>=0.19
   - pip
+  - pseudonetcdf>=3.0.1
+  - pydap
+  # - pynio  # Not available on Windows
+  - pytest
+  - pytest-cov
+  - pytest-env
+  - rasterio
   - scipy
   - seaborn
+  - sparse
   - toolz
-  - rasterio
-  - boto3
   - zarr
+  - pip:
+    - numbagg

ci/requirements/py37-windows.yml

@@ -3,34 +3,40 @@ channels:
   - conda-forge
 dependencies:
   - python=3.7
+  - black
+  - boto3
+  - bottleneck
+  - cdms2
+  - cfgrib>=0.9.2
   - cftime
-  - nc-time-axis
+  - coveralls
   - dask
   - distributed
-  - h5py
+  - flake8
   - h5netcdf
+  - h5py
+  - hypothesis
+  - iris>=1.10
+  - lxml
   - matplotlib
-  - netcdf4
-  - pytest
-  - pytest-cov
-  - pytest-env
-  - coveralls
-  - flake8
   - mypy==0.720  # Must match .pre-commit-config.yaml
+  - nc-time-axis
+  - netcdf4
+  - numba
   - numpy>=1.12
   - pandas>=0.19
   - pip
+  - pseudonetcdf>=3.0.1
+  - pydap
+  - pynio
+  - pytest
+  - pytest-cov
+  - pytest-env
+  - rasterio
   - scipy
   - seaborn
   - sparse
   - toolz
-  - rasterio
-  - boto3
-  - bottleneck
   - zarr
-  - pseudonetcdf>=3.0.1
-  - cfgrib>=0.9.2
-  - lxml
-  - pydap
   - pip:
     - numbagg

ci/requirements/py37.yml

@@ -3,7 +3,7 @@
 This directory contains property-based tests using a library
 called [Hypothesis](https://github.com/HypothesisWorks/hypothesis-python).
 
-The property tests for Xarray are a work in progress - more are always welcome.
+The property tests for xarray are a work in progress - more are always welcome.
 They are stored in a separate directory because they tend to run more examples
 and thus take longer, and so that local development can run a test suite
 without needing to `pip install hypothesis`.

properties/README.md

@@ -4,6 +4,10 @@
 These ones pass, just as you'd hope!
 
 """
+import pytest
+
+pytest.importorskip("hypothesis")
+
 import hypothesis.extra.numpy as npst
 import hypothesis.strategies as st
 from hypothesis import given, settings
@@ -23,6 +27,7 @@
 )
 
 
+@pytest.mark.slow
 @given(st.data(), an_array)
 def test_CFMask_coder_roundtrip(data, arr):
     names = data.draw(
@@ -36,6 +41,7 @@ def test_CFMask_coder_roundtrip(data, arr):
     xr.testing.assert_identical(original, roundtripped)
 
 
+@pytest.mark.slow
 @given(st.data(), an_array)
 def test_CFScaleOffset_coder_roundtrip(data, arr):
     names = data.draw(

properties/test_encode_decode.py

@@ -3,7 +3,7 @@ universal = 1
 
 [tool:pytest]
 python_files=test_*.py
-testpaths=xarray/tests
+testpaths=xarray/tests properties
 # Fixed upstream in https://github.com/kwgoodman/bottleneck/pull/199
 filterwarnings =
     ignore:Using a non-tuple sequence for multidimensional indexing is deprecated:FutureWarning

setup.cfg

@@ -3487,7 +3487,10 @@ def test_uamiv_format_write(self):
             "example.uamiv", engine="pseudonetcdf", backend_kwargs=fmtkw
         )
         with self.roundtrip(
-            expected, save_kwargs=fmtkw, open_kwargs={"backend_kwargs": fmtkw}
+            expected,
+            save_kwargs=fmtkw,
+            open_kwargs={"backend_kwargs": fmtkw},
+            allow_cleanup_failure=True,
         ) as actual:
             assert_identical(expected, actual)
 

xarray/tests/test_backends.py

@@ -656,7 +656,7 @@ def test_dataarray_repr(self):
         a = xr.DataArray(
             sparse.COO.from_numpy(np.ones(4)),
             dims=["x"],
-            coords={"y": ("x", sparse.COO.from_numpy(np.arange(4)))},
+            coords={"y": ("x", sparse.COO.from_numpy(np.arange(4, dtype="i8")))},
         )
         expected = dedent(
             """\
@@ -671,7 +671,7 @@ def test_dataarray_repr(self):
     def test_dataset_repr(self):
         ds = xr.Dataset(
             data_vars={"a": ("x", sparse.COO.from_numpy(np.ones(4)))},
-            coords={"y": ("x", sparse.COO.from_numpy(np.arange(4)))},
+            coords={"y": ("x", sparse.COO.from_numpy(np.arange(4, dtype="i8")))},
         )
         expected = dedent(
             """\