Need to call `asyncio.set_event_loop()` in my tests

[AndreLouisCaron]

Hi there!

I'm porting some of my tests from v0.3.0 to v0.5.0 and I have a test that behaves strangely when I switch to v0.4.1 or later.

After some debugging, I noticed that id(event_loop) inside the test and id(asyncio.get_event_loop()) return different results for versions after v0.4.1 and v0.5.0.

After adding a call to asyncio.set_event_loop(event_loop) in my tests like this:

def test_serve(event_loop, cli):
    asyncio.set_event_loop(event_loop)  # wasn't required with v0.3.0.

Notice that I'm not using @pytest.mark.asyncio here. I can't use it because I'm testing a click CLI that calls event_loop.run_until_complete() internally.

I guess this is explained by the fact that the event_loop fixture stopped calling asyncio.set_event_loop() when the @pytest.mark.asyncio(forbid_global_event_loop=...) syntax was introduced in v0.4.0.

In short, it seems like using only the event_loop fixture (e.g. without @pytest.mark.asyncio) leads to a surprising result. I thing that the (absence of a) relationship between the event_loop fixture and asyncio.get_event_loop() should be added to the documentation to make it more obvious. If you're OK with this, I can send in a PR to fix it.

In passing, I think there's a big quirk in the API here: the behaviour for using the event_loop fixture without the asyncio marker is surprising, to say to least. The havior in v0.3.0 was much better than it is now in that respect. This seems to be caused by the fact that forbid_global_event_loop is marker parameter rather than a fixture. If the policy for controlling the relationship with the default event loop were a fixture, it would allow the event_loop fixture to decide what to do with the default event loop.

I've seen @asvetlov's comments in issue #12 and PR #24. There seems to be some contention over whether returning exceptions from asyncio.get_event_loop() should be the default or not. I don't have a specific position on this, but I am convinced that either it should return the same value as the event_loop fixture or it should fail (having two event loops by default is not ideal). A neat side-effect of making the policy for the default event loop a fixture on which the event_loop fixture depends (instead of a parameter to the marker) would be to allow defining a project-specific default: each project would be able to define this new fixture as something returning a boolean (or enum value) to choose the policy and use autouse to make it a project specific default.

What are your thoughts on this?

[Tinche]

Hello!

You're basically correct in your assessment. The event_loop fixture is really simple, and that's so it can be overridden easily and with basically no knowledge of even pytest. Personally I think wanting to use a faster loop, like uvloop, is a really common scenario even for beginners.

The fact of the matter is, if your test isn't marked pytest.mark.asyncio it's not a pytest-asyncio test. It's just a normal test. You happen to be using a fixture we define. Without the marker, no special behavior will trigger and that's by design.

Now, as for the event loop policy. I've considered making it a fixture but went with a pytest fixture hook wrapper instead. This hook wrapper is, again, only set to execute if the test is marked asyncio. This spares me explaining to users that, sometimes, if they want to have their fixtures depend on the event loop or want to invoke event_loop.run_until_complete themselves, their fixtures/tests really need to depend on the event_loop_policy fixture instead.

Having the event_loop fixture depend on another fixture, like an event_loop_policy_status fixture is interesting but I think ultimately too complex.

I'm basically happy with this arrangement. Your use case almost doesn't require pytest-asyncio at all, you can get away with just defining two fixtures which are both basically one-liners. I'd suggest you just create an autouse fixture to set the event loop into the policy.

I won't say no to improving the docs, of course. :)

[AndreLouisCaron]

Your use case almost doesn't require pytest-asyncio at all,

Except most tests in this project use your asyncio maker. I have a few tests that run without it to check for regressions in my CLI and these tests cannot use the marker (as previously explained, this would require .run_until_complete() to be reentrant).

You happen to be using a fixture we define. Without the marker, no special behavior will trigger and that's by design.

Using the fixture without the marker leads to hard to debug issues related to having two different event loops active at the same time in the same context. That's a feature?

This spares me explaining to users that, sometimes, if they want to have their fixtures depend on the event loop or want to invoke event_loop.run_until_complete themselves, their fixtures/tests really need to depend on the event_loop_policy fixture instead.

I'm not sure I follow you. What I proposed is essentially having the original event_loop fixture from 0.3.0, but to have it depend on a second fixture called forbid_global_event_loop.

@pytest.fixture(scope='session')
def forbid_global_event_loop():
    return False  # Can be True, I really don't mind.

# Not tested :-)
@pytest.yield_fixture(scope='session')
def event_loop(forbid_global_event_loop):
    loop = asyncio.new_event_loop()
    if forbid_global_event_loop:
        asyncio.set_event_loop_policy(ForbiddenEventLoopPolicy())
    yield loop
    loop.close()

That way, a project can simply put this in their conftest.py to decide whether they want to allow their code to use the default event loop or not:

@pytest.fixture
def forbid_global_event_loop():
    return True

The event_loop fixture is really simple, and that's so it can be overridden easily [...] Personally I think wanting to use a faster loop, like uvloop, is a really common scenario even for beginners.

This, I wholeheartedly agree with. I need to support Windows a lot and since the default SelectorEventLoop for Windows is a really inconvenient default, all my projects need to switch to ProactorEventLoop as soon as I write the first test for anything that uses asyncio. Rewriting the entire event_loop just to pick the event loop class would be a PITA.

Maybe this requires more thinking, but my first reflex would be to allow people to provide their own event loop factory without having to rewrite the event_loop fixture. You can extend my proposal above to allow people to supply their custom event loop class with a one-liner like this:

@pytest.fixture
def new_event_loop():
    return UVLoop()  # Whatever.

And then reference this from your event_loop fixture to create the new event loop instead of calling asyncio.new_event_loop().

This way:

1. you keep the one-liner event loop override;
2. @asvetlov gets an easy way to forbid global event loop access for his entire project with a one-liner; and
3. I get to keep your event_loop fixture even for my tests that don't use the marker.

[Tinche]

There already is an event loop factory fixture. It's just called event_loop. :) I believe mutating global state is a tricky thing; the event_loop fixture just makes an event loop for you and hands it over; closes it afterwards.

I considered your idea while I was preparing the 0.5 release, but I find the extra fixtures inelegant. (Especially fixtures that yield configuration.) Anyway, I'm not interested in discussing the subtleties of API design. What we can discuss is how to make the current design work for some of these cases.

For example, the issue of repeating yourself a lot to forbid the global event loop could be solved by http://doc.pytest.org/en/latest/example/markers.html#marking-whole-classes-or-modules. I actually thought that would work out of the box, but alas, no: pytest-dev/pytest#1943

Your problem is that you can't apply the marker to a non-coroutine. So maybe we could make it so the plugin only runs coroutines in the event loop, or make a new marker that would do so.

[AndreLouisCaron]

Your problem is that you can't apply the marker to a non-coroutine. So maybe we could make it so the plugin only runs coroutines in the event loop, or make a new marker that would do so.

Yeah, I guess that would work, but I find it gives a quirky API. I'll end up writing my own pytest plug-in that behaves the way I want it to.