Description
I have some type errors in my code, and would like to find them by running mypy via pytest-mypy. When I run pytest, I get:
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.6.9, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- <venv>
cachedir: .pytest_cache
rootdir: <project-root>, configfile: setup.cfg, testpaths: proof_of_concept, tests
plugins: cov-2.11.1, pycodestyle-2.2.0, pydocstyle-2.2.0, mypy-0.8.1
collected 154 items
proof_of_concept/__init__.py::mypy PASSED
proof_of_concept/__init__.py::mypy-status FAILED
================================================================================= FAILURES =================================================================================
_______________________________________________________________________________ test session _______________________________________________________________________________
mypy exited with status 1.
This suggests that there were some errors, but it doesn't show what they were, so I cannot fix them. Running mypy separately reveals a bunch of issues in proof_of_concept/rest/registry_client.py.
Looking at the pytest-mypy source, it seems that this can happen if mypy checks a file that has not been collected. I'm importing the problem file from the file under test, and mypy gives a bunch of errors about this imported file. I guess the ::mypy test passes anyway, because the errors are not in the file under test. However, pytest-mypy then generates an error using this second mypy-status result, because mypy returned a non-zero exit code.
Ignoring the errors makes sense I think, they should show up later when the problem file itself is tested. Unfortunately, we never get to that point, because I'm running with -x, which stops the tests as soon as the first error is found, and no error messages are shown for the mypy-status failure, so I cannot fix them.
I'm a bit confused about the current intent of the pytest-mypy code. There's a comment suggesting that the ::mypy test is intended to fail even if mypy only shows errors in files that weren't collected, and that that is why there's the ::mypy-status result. On the other hand, there used to be a test that checked that no MypyStatusItem was created unless there was at least one MypyFileItem, which I think means the opposite. That was removed in cd459a4 however, which also introduced this -status result.
In my opinion, a mypy failure only because of errors in imported files should either be considered an error in the file under test, in which case the error should be printed, or it should not be considered an error in the file under test, in which case all tests related to that file should pass and the non-zero exit code ignored. The latter makes more sense to me.