Testing of Security Manager checklist

[alisevych]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• Choose appropriate workflow from the list - with Security Manager included - plugin build used
• Download plugin
• Open IntelliJ IDE
• Remove previously installed UTBot plugin
• Install the downloaded plugin

Go through manual scenarios

Basic security

• Project with JDK 8 + manual scenario from PR Add SecurityManager support to block suspicious code #622 #625 + default settings
• Fuzzing only
• Symbolic execution only
• Project with JDK 11 + default settings, bug found: Socket connect should be sandboxed #792
• Fuzzing only
• Symbolic execution only
• Concrete executor is invoked from Symbolic execution

Project with JDK 17 - blocked as UTBot is incompatible with JDK 12+

Permissions

• add PropertyPermission in ~.utbot\sandbox.policy - check test for property is not in sandbox, execute
• add "permission java.io.FilePermission "<>", "read";" - check test for read not in sandbox, execute
• add AllPermission in ~.utbot\sandbox.policy - check NO tests are in sandbox, try to execute (carefully!!!)
• corrupt permission - check how UTBot fails - found Too many permissions file rereads - can slow down much #796
• empty file
• remove ~.utbot\sandbox.policy file - check generated tests are in sandbox

Test rendering

• JUnit4
• JUnit5
• TestNG
• JUnit5 with parametrization - JUnit5 params dependency is required in default IDEA's Gradle project #620 reproducing
• TestNG with parametrization

Code structures

• in private method
• in static block of a used class - bug found: Implicit unpermitted operations should be sandboxed #791
• in compiler of a used class - bug found: Implicit unpermitted operations should be sandboxed #791
• in a separate thread - not yet supported - enhancement: Disable tests with unpermitted operations in Threads #895

[alisevych]

@Markoutte @korifey
Please advise if UTBot needs to detect the restricted operations in Threads?
The following method is not executed - AccessControlException is thrown (in logs) - that's Ok.
But the generated tests are not disabled due to Sandbox

public int read() throws Throwable {
        Thread t = new Thread("fileRead") {
            public void run()  {
                try {
                    new File("a.txt").createNewFile();
                } catch (IOException e) {
                    throw new RuntimeException(e);
                }
            }
        };
        t.start();
        Thread.sleep(500);
        return 10;
    }