asyncio_mode=auto does not support hypothesis without marker

[TheKevJames]

The new asyncio_mode=auto mode does not support hypothesis markers. I'm assuming this is an issue on the asyncio side, since strict mode continues to work, but please let me know if I should post this to hypothesis instead!

I think @asvetlov and @seifertm are the relevant folks to ping for this.

Reproduction case:

pytest.ini

[pytest]
asyncio_mode = auto

test.py

from hypothesis import given
from hypothesis import strategies as st

@given(x=st.booleans())
async def test_foo(x):
    assert x

» pip install pytest-asyncio hypothesis
» pytest test.py
== test session starts =
platform darwin -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /private/tmp/testpyasynchypo, configfile: pytest.ini
plugins: hypothesis-6.35.0, asyncio-0.17.0
collected 1 item                                                                                                                                                                                                            

test.py F                                                                       [100%]

== FAILURES ==
_ test_foo _

    @given(x=st.booleans())
>   async def test_foo(x):
E   hypothesis.errors.InvalidArgument: Hypothesis doesn't know how to run async test functions like test_foo.  You'll need to write a custom executor, or use a library like pytest-asyncio or pytest-trio which can handle the translation for you.
E       See https://hypothesis.readthedocs.io/en/latest/details.html#custom-function-execution

test.py:6: InvalidArgument
== short test summary info ==
FAILED test.py::test_foo - hypothesis.errors.InvalidArgument: Hypothesis doesn't know how to run async test functions like test_foo.  You'll need to write a custom executor, or use a library like pytest-asyncio or pyte...
== 1 failed in 0.22s ==

Comparison, using strict mode, the following works as expected:

pytest.ini

[pytest]
asyncio_mode = strict

test.py

import pytest
from hypothesis import given
from hypothesis import strategies as st

@pytest.mark.asyncio
@given(x=st.booleans())
async def test_foo(x):
    assert x

[seifertm]

Thanks for the report! I think pytest-asyncio is the correct place to file this issue.

The --asyncio-mode=auto marks all async functions with the asyncio mark during the collection phase. However, when pytest collects the Hypothesis test, the @given decorator has already been applied and the function is no longer a coroutine.

[TheKevJames]

Hmm, yeah, that makes sense. One thing which confuses me a bit on that, though, is that it seems like both of the following work, ie. that order does not matter between the decorators:

@given(...)
@pytest.mark.asyncio
async def test_foo(...):
  ...

@pytest.mark.asyncio
@given(...)
async def test_foo(...):
  ...

I haven't looked at pytest-asyncio code at all, but coming strictly from an understanding of Python it seems to me that the latter case (eg. the asyncio mark wrapping the hypothesis decorator) should have the same order of operations -- ie. that the mark.asyncio would only be applied after hypothesis' decorator has been applied. Perhaps I'm missing something here.

---

EDIT: oh, right, "collection phase" is an entirely different later step, AST parsing is irrelevant 🤦

[seifertm]

@TheKevJames There is no "natural" or correct order of the @pytest.mark.asyncio and @given decorators. Therefore, both pytest-asyncio and hypothesis try to be smart and allow both orderings.

[asvetlov]

An interesting use case, need to investigate.
I know what the hypothesis tool is but never is it personally and never dug into @given decorator implementation.

[TheKevJames]

@seifertm really appreciate the fast turnaround on this -- thank you!