CI: Speed up coverage tests by splitting files

[datapythonista]

In #28531 we started to parallelize tests in the CI by file. Meaning that each file in pandas/tests/ will be assigned to a core, and will run fully there. Before, each individual test was assigned to a core. This makes things much simpler, since tests that cannot run in parallel just need to live in the same file. But it implies, that if a test file is big and contains many tests, it will slow down the running of all tests.

As an example, imagine we have 1,000 tests that take one second each, and we have 100 cores. 10 tests will be assigned to each core, and the test suite will run in 10 seconds using the 100 cores. But with the new set up, if we have a file with 100 tests, all them will run in a single core, and the test suite will take the 100 seconds that the core will need to run the 100 tests.

In practice, after #28531 most of the builds in the CI didn't increases much the time to run. Only the coverage build is in some cases taking significantly more (from around 30 minutes to around 45 minutes).

The solution to this is:

• See how long each test file takes
• Split the files taking longer in a meaningful way

Note that tests can be skipped if dependencies are not found. For example, if matplotlib is not installed in the environment, the plotting tests will simply be skipped. So, it's needed to create an environment with the dependencies of the build ci/deps/travis-36-cov.yaml.

To call the exact tests as the build, see the call in run_tests.sh, and replace the parameters by the ones defined in the build.

[Tirore96]

Hi I'd like to work on this issue

[Tirore96]

In regards to running run_test.sh I got two questions.

1. The parameter values specified in build don't adress $TEST_ARGS and $COVERAGE. What values should these have?
2. When running pytest it reports an "unrecognized arguments: --strict-data-files" error. -strict-data-files is specified as an addopt in the config file. Looking at the options page for pytest I can't find --strict-data-files as an addopt. Am I misssing something?

[datapythonista]

1. $COVERAGE is usually empty, except for the job that computed and uploads the coverage, then it's true: https://github.com/pandas-dev/pandas/blob/master/.travis.yml#L47 Similar for $TEST_ARGS, not sure which job uses it, but it's usually empty, and some build will use it to provide extra args to pytest (if I'm not wrong).
2. It looks like you don't have the right dependencies or the right version of pytest. Use conda for it, in the root of pandas run conda env update if you already created the conda environment, conda env create if you didn't. Make sure you run conda activate pandas-dev after that.

[datapythonista]

To work in an issue, you can write take in a comment and it'll be assigned to you. Please unassign it if you stop working on it. Thanks!

[Tirore96]

Thanks, I'll remember that ;)

[lambertsbennett]

I'd love to take a stab at this. I'll take this issue and try to get it sorted.

[datapythonista]

Thanks @lambertsbennett. I assigned you the issue, you can do it yourself next time by writing a comment with just the word take. If you can't finally finish this would be great if you unassign yourself from the issue, so someone else can give a try. Thanks!

[lambertsbennett]

Hi, sorry to ask, but it's my first time trying something like this. I carried out the following steps:

1. Forked and cloned pandas.
   2)Navigated to ci/deps/ and created an environment with Anaconda using the file travis-36-cov.yaml.
2. Activated the environment.
3. Ran python setup.py build_ext --inplace -j 4
4. Ran pip install -e .
5. Navigated to ci/
6. Set variable PATTERN="((not slow and not network) or (single and db))" as indicated in the build file.
7. Set variable COVERAGE="true" as indicated in the build file.

When I execute run_tests.sh I get the same error Tiore96 indicated above. When I run the pytest command alone in the directory above, the tests will run for some time and then begin to throw many errors. I'm not sure if I'm doing something fundamentally stupid here.