Make it easier to run large number of serial tests

[0xTim]

Description

Something that's cropped up recently and open for discussion!

There are a number of scenarios where you might need to run tests serially, especially when it comes to integration tests (for example running tests against a real database). Currently, with Swift Testing you only have two options to accommodate this, both of which have drawbacks:

• run test tests with the --no-parallel - this is problematic if you have different tests you want to run as well that can be run in parallel
• create a test suite with the .serialized trait - if you have lots of tests you end up with a giant test file that's difficult to maintain. You can still have multiple suites that are marked as serialized, but whilst the tests inside them will be serial, the suites themselves will not, causing clashes.

It would be great to have a better solution to handle use cases like this, either some kind of new trait or flag -no-parallel-serial-tests etc.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[grynspan]

Could you help us to understand the scenarios you're thinking of?

[0xTim]

Sure, imagine you have a Vapor app using the repository pattern. Most of the tests only test business logic without any external dependencies so can be run in parallel without any issue. The integration tests for the database layer however all touch a real database so need to be run in serial otherwise those tests could overwrite one enough.

Currently there isn't an ideal solution to run these tests. You either need to run the integration tests separately with --no-parallel or put them all in a single file. Ideally I'd be able to mark a number of suites as part of the integration tests so only one of those suites can run at once.

(Correct me if I'm wrong but I can't extend a type that's annotated with the @Suite macro to add more tests to that suite in a separate file right?)

[grynspan]

Extensions to suite types are permitted—you just can't put the @Suite attribute on an extension because it becomes ambiguous:

@Suite("My name is Batman!") extension S {}

@Suite("No, it's Mr. Freeze!") extension S {}

But you can absolutely put tests in extensions to suites, or even nest them. So you could have:

@Suite(.serialized) struct DatabaseTests {}

extension DatabaseTests {
  @Suite struct ReadingTests {
    ...
  }
}

extension DatabaseTests {
  @Suite struct WritingTests {
    ...
  }
}

And all three top-level declarations (the suite type and its two extensions) can go in separate files.

I hope that helps!

[0xTim]

Oh awesome - so just to confirm ReadingTests and WritingTests would never be run at the same time? If that's the case I'm happy to close the issue!

[stmontgomery]

To document that pattern better, I filed #686

[grynspan]

Oh awesome - so just to confirm ReadingTests and WritingTests would never be run at the same time? If that's the case I'm happy to close the issue!

Correct—everything inside DatabaseTests in this example will run one… test… at… a… time, and parameterized tests will run one… case… at… a… time. :)

If you find that is not the case, that's a bug, so please let us know!

[stmontgomery]

Given that we believe this pattern already exists and works, let's close this. Feel free to reopen if you find that it isn't working as described.