Test with comment 'This test fails because ... java.lang.ArithmeticException' does not fail

[tyuldashev]

Description

UTBot could find situations which leads to arithmetic overflow and generate tests for such cases adding comment that such test would fail, but actually such test pass.

To Reproduce

Steps to reproduce the behavior:

1. Open utbot project in IntelliJ Idea
2. In UTBot settings set Overflow detection: to Treat overflows as errors
3. Set Test generated method to 100 % Symbolic
4. Generate tests for org.utbot.examples.math.OverflowExamples#intOverflow method
5. Open the generated test

Expected behavior

Either UTBot generates test which causes ArithmeticException during execution
Or it doesn't write that test should fail

Actual behavior

Test which are supposed to generate errors are generated, but during execution they pass.

Sample of such test:

    @Test
    @DisplayName("intOverflow: x * x * x > 0 -> ThrowArithmeticException")
    public void testIntOverflow_ThrowArithmeticException() {
        OverflowExamples overflowExamples = new OverflowExamples();

        /* This test fails because method [org.utbot.examples.math.OverflowExamples.intOverflow] produces [java.lang.ArithmeticException: int  *  overflow] */
        overflowExamples.intOverflow(-11086, 0);
    }

Visual proofs (screenshots, logs, images)

Additional context

So this case is more about how to properly present results of UTBot finding. Currently there are at least two problems:

• incorrect comment that test fails
• as test pass it would be difficult to find such test in case of bulk test generation, so overflow problem could be left unnoticed.
  Probably we should add fail() to such test, so it would actually fail and draw user attention.

[denis-fokin]

Not enough resources, move to the next release