Make default xfail strict

[TomAugspurger]

Pytest let's you configure the default xfail strictness in the setup.cfg

xfail_strict=true

This issue is to

1. Make that change
2. Update tests that are currently XPASSing
   • If always passing then remove the xfail entirely
   • If only sometimes passing (flaky test, dependency, configuration, etc.) then mark that specific test as strict=False.

cc @jbrockmendel @jreback

[kaushik94]

Hi, I am a beginner looking to contribute to pandas. Would like to learn a bit from this issue, can I take this up?

[TomAugspurger]

Sure.

I would run something like pytest pandas -ra which should print out all the tests that xfail and xpass. From there, you can go to the individual tests that are xpass and figure out what needs to be done.

Also, there look to be ~100 xfail tests (not sure how many are not strict). If there are too many then feel free to do fixes in batches and just fix a subset of the files.

[kaushik94]

Ok, sounds good. Will check it out and get back to you soon.

[cgangwar11]

If this issue is not yet resolved can I work on this

[dcreekp]

hello, I've made a pull request
#23721
but the travis-ci checks keep failing. Any help would be appreciated.
The error message mentions this: #21636
and another error points to:

=================================== FAILURES ===================================
_______________________ pandas/tests/series/test_rank.py _______________________
[gw0] linux2 -- Python 2.7.15 /home/travis/miniconda3/envs/pandas/bin/python
Worker 'gw0' crashed while running 'pandas/tests/series/test_rank.py::test_pct_max_many_rows'
 generated xml file: /home/travis/build/pandas-dev/pandas/test-data-multiple.xml 

[jbrockmendel]

The test_pct_max_many_rows error is unrelated, don't worry about it for now.

_______________________________ test_apply[True] _______________________________
[gw0] linux -- Python 3.7.1 /home/travis/miniconda3/envs/pandas/bin/python
[XPASS(strict)] flaky on 3.7, xref gh-21636
______________________________ test_apply[False] _______________________________
[gw0] linux -- Python 3.7.1 /home/travis/miniconda3/envs/pandas/bin/python
[XPASS(strict)] flaky on 3.7, xref gh-21636

This indicates that the test test_apply (it's a shame it doesn't tell you the file where the test is located, but I'm pretty sure its tests/groupby/test_categorical.py) is passing despite being marked as an xfail. It either needs to be un-xfailed (if it is passing consistently for all builds) or have the strict tag removed.