Add testutils to mark tests as flaky

[fendor]

Issue: #3581

This PR does three things:

• Don't rerun CI test failures
• Introduce combinators to mark tests as flaky
• Mark all tests as flaky that fail on CI

[July541]

Should we have a corresponding ticket for every flaky test?🤔️

It would be great if every knowFlaky has a github issue id in the code.

[fendor]

Should we have a corresponding ticket for every flaky test?

It would be great, but looking at this CI result, I expect to have 40-50 flaky tests.
Having an issue for each of them looks like overkill to me. However, I think we should have an issue per package/plugin that has flaky tests. Then maybe maintainers also feel more inclined to look at their test suites.

[July541]

Aha, 40-50 really exceeds my expectations, issues for every plugin are reasonable...

[fendor]

Due to the extreme flakiness of lsp-tests, I'm back paddling my approach of marking everything that fails once as flaky. That's a bit disappointing, but I think we'd have to essentially mark every test as flaky eventually, since it is difficult to write a completely deterministic test with lsp-test (not necessarily the fault of lsp-test).
Thus, we run the test suite twice to wheat out the most flaky behaviour. If a test fails twice in a row, it is marked as flaky. It looks like this will allow us to have fewer flaky tests, thus creating one issue for each seems feasible again.

[fendor]

Closing as this seems to be useless for now and I don't intend to get this ready for merge any time soon.