Summarizer erases JVM crash executions

[dtim]

Description

When the concrete executor tries to execute a branch that leads to a JVM crash, it creates a UtExecution instance that correspond to the failure. The test generator will process this execution in a special way: the generated test case will be disabled and it will contain a comment about the possible JVM crash.

Summarizer considers only two subclasses of UtExecution, UtSymbolicExecution and UtFuzzedExecution, but not the parent class itself. As a result, summarizer erases these failed (JVM crash) executions, and no tests are generated for them.

A related problem: failed executions are not erased during the code generation / summarization phases in utbot-framework unit tests, because the result of the call to summarizer is not assigned to anything. It seems that this inconsistency should be fixed as well.

To Reproduce

Turn fuzzing off (either via the plugin settings dialog window or by setting UtSettings option useFuzzing to false).

Generate a test suite for the following class using the plugin.

import sun.misc.Unsafe;

import java.lang.reflect.Field;
import java.security.AccessController;
import java.security.PrivilegedAction;

public class JvmCrashExample {
    public int crashPrivileged(int i) {
        return AccessController.doPrivileged((PrivilegedAction<Integer>) () -> {
            try {
                Field unsafeField = Unsafe.class.getDeclaredField("theUnsafe");
                unsafeField.setAccessible(true);
                Unsafe unsafe = (Unsafe) unsafeField.get(null);
                unsafe.putAddress(0, 0);

                if (i == 0) {
                    return i;
                }

                return 1;
            } catch (NoSuchFieldException | IllegalAccessException e) {
                throw new RuntimeException(e);
            }
        });
    }
}

Note: this example uses doPrivileged to allow the concrete executor to run privileged code (the unsafe call) despite the sandbox limitations. If the crashing code is not wrapped in doPrivileged, and the operation is not explicitly enabled by a sandbox configuration file, a special test case will be generated instead with the annotation @Disabled(value = "Disabled due to sandbox"). To reproduce this issue, please either allow all unsafe operations to the concrete executor, or use doPrivileged.

Expected behavior

A disabled test case should be generated.

import org.junit.jupiter.api.Test;

public class JvmCrashExampleTest {
    ///region Test suites for executable unsafe.JvmCrashExample.crashPrivileged

    ///region

    /**
     * <a href="file:////home/dtim/IdeaProjects/Jps11/hs_err_pid18234.log">JVM crash report</a>
     */
    @Test
    @org.junit.jupiter.api.Disabled(value = "Disabled due to possible JVM crash")
    public void testCrashPrivileged1() {
        JvmCrashExample jvmCrashExample = new JvmCrashExample();

        // This invocation possibly crashes JVM
        jvmCrashExample.crashPrivileged(-255);
    }
    ///endregion

    ///endregion

}

This test case is successfully generated when UtSettings.enableMachineLearningModule is set to false (i.e., summaries are disabled).

Actual behavior

When UtSettings.enableMachineLearningModule is set to true (default value, summaries are enabled), no test cases are produced, and the plugin shows the usual "no tests generated" message box.

Environment

This issue can be reproduced with Java 11 as the project JDK, although it is not JDK-specific. It does not depend on mocking settings or JUnit/TestNg version as well.

Additional context

I'd suggest to fix this issue from both sides:

• UtExecution should be converted to a sealed class, and a dedicated child class (e.g., UtFailedExecution) should be used to represent JVM failures instead of plain UtExecution.
• Summarizer code should be adjusted to correctly handle all execution types (symbolic executions, fuzzer executions, and failure executions) to avoid accidently dropping any of them.

The summary generation phase in the utbot-framework unit tests should be fixed as well to match the plugin behavior.

[amandelpie]

@dtim super, will do next week

[dtim]

@amandelpie Thank you. I am willing to prepare the engine part if you agree with the overall approach (let's discuss it next week).

[CaelmBleidd]

@amandelpie do we have this problem now? Or we should close it since there is no problem anymore

[alisevych]

@amandelpie, I've checked on the latest main - it's not reproducing. So closing as stale.