New `LoadScheduling` batch distribution logic doesn't work for long-running tests

[mkoura]

The PR #812 that implemented new batch distribution logic (and that is now released in 3.0.2) disregards long-running tests.

I have several modules with long-running tests and many of these tests are now scheduled on single worker, instead of being distributed evenly across multiple workers. As a result, total runtime of my testsuite more than doubled from ~ 2.5 hours to 5+ hours.

Are there any workarounds to bring back the original round-robin scheduling?

cc @amezin

[mkoura]

It looks like using the LoadGroupScheduling (--dist loadgroup) can be used as workaround.

[amezin]

Hm... The simplest solution would be to add another scheduler class, derived from LoadScheduling, where initial batch is distributed round-robin.

However, it will help only if your slow tests will fit into that initial batch (although if the difference is so huge - looks like previously they did).

Another option is to significantly change how LoadScheduling works. I have a few ideas, but not sure how they will turn out in reality.

Is the test suite you're talking about publicly available?

[amezin]

Or, maybe, a more flexible solution: limit size of the chunks of consecutive tests that xdist submits to workers (in ini file and/or CLI argument). Setting it to 1 will effectively mean "round-robin distribution".

[mkoura]

@amezin Good idea with the config option.

My testsuite is public - https://github.com/input-output-hk/cardano-node-tests/tree/master/cardano_node_tests/tests
I'm using pytest-order to make sure the long-running tests fit into initial batch.

[mkoura]

I've solved this for myself with custom scheduler. It schedules long-running tests as early as possible while avoiding scheduling long-running tests one after the other (my long-running tests are marked with pytest.mark.long). My testsuite now finishes in ~ 1:15 hours, which is runtime of the longest test.

@amezin The work stealing scheduler is great idea for general purpose scheduler.

The issue can be closed if you don't want to track my usecase.

[nicoddemus]

I think #857 will add some builtin support as a workaround, but the custom scheduler is indeed the optimal solution as it can be adapted for your specific test suite.

I will be closing this then, thanks everyone for the discussion.

[amezin]

@mkoura out of curiosity, could you disable reordering workarounds, and try pytest-xdist master branch with --dist worksteal?