Order in which tests are executed

[blueyed]

pytest-django could / should probably use the same test ordering like Django: https://docs.djangoproject.com/en/1.7/topics/testing/overview/#order-in-which-tests-are-executed

This would mean to run all tests using the db fixture before tests using the transactional_db fixture, and then the remaining ones.

Django's tests should be run according to the Django documentation linked above.

This could be achieved using pytest's pytest_collection_modifyitems probably.

Some related pytest plugin: https://github.com/ftobia/pytest-ordering

[pelme]

I don't see very strong reasons to reorder the tests, but I'm not really against it either if it is easy to pull of.

I'm however strongly in favor of having tests that can run in any order. In my opinion, tests that doesn't clean up after themselves or tests that rely on state from other tests is broken and I think it is fine for pytest-django to be incompatible from Djangos test runner in those cases.

Here are some background discussions about ordering of tests in Django:
https://groups.google.com/forum/#!msg/django-developers/PfK0km66CNI/JbReB0pDkR8J
https://code.djangoproject.com/ticket/18271

[blueyed]

See #223.

I'm however strongly in favor of having tests that can run in any order.

This could be done / tested using https://github.com/klrmn/pytest-random, which uses the same pytest hook and therefore order probably matters when used with the patch from #223.

[blueyed]

Maybe we could add pytest-random (or the idea from there) to our Travis test suite?

[pelme]

The pytest-django test suite itself is probably not really random safe - there are some tests that kind of expects order (i.e. all test_*_again tests). I regard these as bugs, and these tests should definitely be replaced with proper testdir.runpytest() tests to make sure they are actually testing the right thing.

I am against re-ordering tests to conform with Django's order, the order should not make any difference with regard to compatibility anyways. I think re-ordering of tests is better left to pytest itself (possible future optimizations by being smarter about fixture caching etc), pytest-random and the pytest-xdist scheduler unless there is a strong reason to change the order.

I feel bad about this since you have put a lot of work into this and created a very nice patch, but I think we're better of by not merging this.

What are your thoughts about this?

[dan-passaro]

I'm personally +1 on this. I agree with you @pelme that, ideally, the order shouldn't matter. But here I am trying to chase down how we have a test that works fine on its own in but breaks when run after another specific test. This extra ramp-up time in getting pytest-django running doesn't look good for me or my decision to try and switch to pytest (for right now anyway, I'm sure once I get it up and running all will be forgiven).

If this pain on my end can be avoided by just taking up the same ordering as Django, I think that's a net win. It'll make it easier for existing Django projects to switch, even if they have mistakes in their test suites which are masked by the order the Django runner is executing their suite.

As un-ideal as it is, it is practical. Even having the option to explicitly allow re-ordering would be nice, that way I can show my team that when run the normal way everything appears good but... shuffle the order around and look at all the hidden bugs we have in our test suite!! Then we can plan time to address and fix that while not having that block our adoption of pytest and pytest-django.

[alvinchow86]

At least, it's not too hard to do your own ordering with pytest_collection_modifyitems.

The application I had for this was that my test suite has some expensive DB fixtures that are set up for every test, and it was an enormous (probably 2-3x) overall performance gain to set up the fixtures first, then run each test (each one is already wrapped in a transaction), then run any tests requiring transactional database (e.g. @pytest.mark.transactional_db) at the end of the test run. Although to get this to work, the test reordering was the easiest thing..

[blueyed]

@dan-passaro
Have you tried #223?

[blueyed]

Looking at #223 it seems like it should also handle Django's test classes, and not only look at the pytest-specific markers.

[bparker98]

We have found that after upgrading to Django 1.8 our tests that inherited from TransactionTestCase were causing our other tests to fail due to the database being rolled back to an empty state after TransactionTestCase finishes.

Adding serialized_rollback = True to our TransactionTestCase instances allowed those tests to run, but when another test that inherited from Django's TestCase ran after running a TransactionTestCase it would fail because the database would be empty again.

Adding serialized_rollback = True to all of our test cases slowed things down so much it was not an option.

So there are side-effects to not maintaining the same order.

Are there best practices we should be looking into to avoid this issue?

[penoux]

+1 on previous comment from @bparker98. We are switching from Django TestRunner to Pytest Django, but we have not yet rewritten all our TestCase and TransactionTestCase to pytest. When Pytest executes our suite, the database is flushed after last TransactionTestCase (with serialized_rollback = True). The following TestCase then fails at each time. Django's strategy (running all TestCase, then all TransactionTestCase) may not be something to follow in the long term, but without it, Pytest brings a lot of regression (in our case though).

[rlucioni]

I was struggling to switch an existing project over to pytest-django because of this issue. I took @blueyed's approach from #223, extended it to support Django's TransactionTestCase subclasses, turned it into a plugin, and put it on PyPI: https://pypi.python.org/pypi/pytest-django-ordering. If you want to use pytest-django but preserve the order in which Django runs tests, you can simply install the plugin alongside pytest-django.

[blueyed]

@rlucioni
Thanks.
Do you think it would be easy to integrate it, based on an option?