Skip to content

Show risky disabled tests as Problems and in SARIF-report #1168

Closed
@alisevych

Description

@alisevych

Description

Generated tests disabled due to Sandbox or due to possible JVM crash are not show in SARIF-report and on Problems tab.

To Reproduce

  1. Run the 'UTBotJava' project in IntelliJ Idea
  2. Both Fuzzing and Symbolic analysis are enabled
  3. UnitTestBot plugin is installed from branch Built-in SARIF reports visualizer #1166: build
  4. In Project view find utbot-sample/src/main/java/org/utbot/examples/exceptions/JvmCrashExamples.java
  5. Generate tests with UnitTestBot...
  6. Check there are tests "Disabled due to possible JVM crash" and "Disabled due to sandbox"

Expected behavior

These found problems need to:

  1. add to error suite
  2. add to SARIF-report
  3. display on Problems tab with the respectful stack trace

Actual behavior

Currently executions that are stopped on AccessControlException - Disabled due to sandbox - are not added to SARIF report as errors and are not displayed on Problems tab.
Same with the tests "Disabled due to possible JVM crash".

Visual proofs (screenshots, logs, images)

image

image

public class JvmCrashExamplesTest {
    ///region Test suites for executable org.utbot.examples.exceptions.JvmCrashExamples.crash

    ///region OTHER: SECURITY for method crash(int)

    @Test
    @org.junit.Ignore(value = "Disabled due to sandbox")
    public void testCrash1() {
        JvmCrashExamples jvmCrashExamples = new JvmCrashExamples();
        
        /* This test fails because method [org.utbot.examples.exceptions.JvmCrashExamples.crash] produces [java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.misc")]
            java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
            java.base/java.security.AccessController.checkPermission(AccessController.java:897)
            java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
            java.base/java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1290)
            java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:174)
            java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:576)
            org.utbot.instrumentation.process.HandlerClassesLoader.loadClass(ChildProcess.kt:48)
            java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
            org.utbot.examples.exceptions.JvmCrashExamples.crash(JvmCrashExamples.java:20) */
    }
    ///endregion

    ///endregion

    ///region Test suites for executable org.utbot.examples.exceptions.JvmCrashExamples.crashPrivileged

    ///region

    @Test
    @org.junit.Ignore(value = "Disabled due to possible JVM crash")
    public void testCrashPrivileged1() {
        JvmCrashExamples jvmCrashExamples = new JvmCrashExamples();

        // This invocation possibly crashes JVM
        jvmCrashExamples.crashPrivileged(1);
    }
    ///endregion

    ///endregion

    ///region Test suites for executable org.utbot.examples.exceptions.JvmCrashExamples.exit

    ///region FUZZER: SUCCESSFUL EXECUTIONS for method exit(int)

    /**
     * @utbot.classUnderTest {@link JvmCrashExamples}
     * @utbot.methodUnderTest {@link JvmCrashExamples#exit(int)}
     */
    @Test
    public void testExitReturnsOne() {
        JvmCrashExamples jvmCrashExamples = new JvmCrashExamples();

        int actual = jvmCrashExamples.exit(1);

        assertEquals(1, actual);
    }
    ///endregion

    ///region OTHER: SECURITY for method exit(int)

    @Test
    @org.junit.Ignore(value = "Disabled due to sandbox")
    public void testExit1() {
        JvmCrashExamples jvmCrashExamples = new JvmCrashExamples();
        
        /* This test fails because method [org.utbot.examples.exceptions.JvmCrashExamples.exit] produces [java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "exitVM.-1")]
            java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
            java.base/java.security.AccessController.checkPermission(AccessController.java:897)
            java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
            java.base/java.lang.SecurityManager.checkExit(SecurityManager.java:534)
            java.base/java.lang.Runtime.exit(Runtime.java:114)
            java.base/java.lang.System.exit(System.java:1752)
            org.utbot.examples.exceptions.JvmCrashExamples.exit(JvmCrashExamples.java:12) */
    }
    ///endregion

    ///endregion
}

Additional context

Originally posted by @alisevych in #1166 (comment)

Metadata

Metadata

Assignees

Labels

comp-uiImprovements of plugin UI appearence and functionalityctg-bugIssue is a bug

Type

No type

Projects

Status

Done

Relationships

None yet

Development

No branches or pull requests

Issue actions