Skip to content

Prefer symbolic executions during minimization #854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

dtim
Copy link
Collaborator

@dtim dtim commented Sep 4, 2022

Description

Suppose we have two executions with the same coverage, one produced by the symbolic engine, and one produced by the fuzzer. From the user's point of view, they are equivalent, and the minimizer can keep any of them. Usually it will be fuzzed one, because fuzzer produces executions earlier that the symbolic engine, and the minimizer orders executions with the same coverage by their order in the execution list.

When parametrized tests are generated, fuzzed tests tend to be excluded due to mock processing issues. If a fuzzed tests has been selected by the minimizer instead of the symbolic one, the corresponding path remains uncovered: one (symbolic) test has been dropped by the minimizer, and another (fuzzed) test has been excluded by the code generator.

The coverage of the entire parametrized test suite may be improved is minimizer selects symbolic executions when possible. It does not guarantee the full coverage (it is possible to have only fuzzed tests for some paths), but we expect that the coverage may be improved that way.

To let users to control the minimizer behavior, new UtSettings option is introduced: preferSymbolicExecutionsDuringMinimization. When it is set to true (default), the minimizer will consider the execution source (specifically, its class: UtSymbolicExecution or UtFuzzedExecution (UtFailedExecution and any potential other direct UtExecution subclasses are assigned the lowest priority, but UtFailedExecution does not have any coverage anyway, so it is processed separately).

Fixes #843

Type of Change

  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

This fix changes the set of generated tests. It should not decrease coverage, and is expected to increase coverage when generating parametrized tests (please report a bug if coverage is decreased when preferSymbolicExecutionsDuringMinimization is true).

This PR also changes the minimizer interface.

How Has This Been Tested?

Automated Testing

All existing minimizer tests should pass.

Two new minimizer tests have been added:

  • org.utbot.framework.minimization.MinimizationGreedyEssentialTest#testWithSourcePriority
  • org.utbot.framework.minimization.MinimizationGreedyEssentialTest#testWithoutSourcePriority

Manual Scenario

Make sure that fuzzer is enabled in UtSettings, and both fuzzer and symbolic engine are enabled in the plugin configuration window.

Create any method simple enough so both fuzzer and symbolic engine can cover all branches. An example:

package simple;

public class SimpleExample {
    public int foo(int a, int b) {
        if (a > b)
            return a - b;
        else
            return a + b;
    }
}

Set UtSettings.preferSymbolicExecutionsDuringMinimization to true and generate a test suite for foo. Test cases produced by the symbolic engine should be generated.

package simple;

import org.junit.Test;

import static org.junit.Assert.assertEquals;

public class SimpleExampleTest {
    ///region Test suites for executable simple.SimpleExample.foo

    ///region SUCCESSFUL EXECUTIONS for method foo(int, int)

    /**
     * <pre>
     * Test executes conditions:
     *     {@code (a > b): False }
     * returns from: {@code return a + b; }
     * </pre>
     */
    @Test
    public void testFoo_ALessOrEqualB() {
        SimpleExample simpleExample = new SimpleExample();

        int actual = simpleExample.foo(-240, -240);

        assertEquals(-480, actual);
    }

    /**
     * <pre>
     * Test executes conditions:
     *     {@code (a > b): True }
     * returns from: {@code return a - b; }
     * </pre>
     */
    @Test
    public void testFoo_AGreaterThanB() {
        SimpleExample simpleExample = new SimpleExample();

        int actual = simpleExample.foo(1, 0);

        assertEquals(1, actual);
    }
    ///endregion

    ///endregion
}

Set UtSettings.preferSymbolicExecutionsDuringMinimization to false and generate a test suite for foo again. This time, the plugin should generate test cases produced by the fuzzer.

package simple;

import org.junit.Test;

import static org.junit.Assert.assertEquals;

public class SimpleExampleTest {
    ///region Test suites for executable simple.SimpleExample.foo

    ///region

    @Test
    public void testFooReturnsZero() {
        SimpleExample simpleExample = new SimpleExample();

        int actual = simpleExample.foo(-1, 1);

        assertEquals(0, actual);
    }

    @Test
    public void testFooWithCornerCase() {
        SimpleExample simpleExample = new SimpleExample();

        int actual = simpleExample.foo(Integer.MAX_VALUE, 1);

        assertEquals(2147483646, actual);
    }
    ///endregion

    ///endregion
}

Checklist (remove irrelevant options):

This is the author self-check list

  • The change followed the style guidelines of the UTBot project
  • Self-review of the code is passed
  • The change contains enough commentaries, particularly in hard-to-understand areas
  • New documentation is provided or existed one is altered
  • No new warnings
  • New tests have been added
  • All tests pass locally with my changes

@dtim dtim force-pushed the dtim/843_minimizer_prefer_symbolic branch from 3dbddc1 to f612a03 Compare September 5, 2022 09:44
@dtim dtim changed the title Prefer symbolic executions during minimization (draft) Prefer symbolic executions during minimization Sep 5, 2022
@dtim dtim force-pushed the dtim/843_minimizer_prefer_symbolic branch 2 times, most recently from d035957 to 3801e27 Compare September 5, 2022 11:36
@dtim dtim marked this pull request as ready for review September 5, 2022 11:36
private fun UtExecution.getSourcePriorityForMinimization(): Int = when (this) {
is UtSymbolicExecution -> 0
is UtFuzzedExecution -> 1
else -> 2
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is about FailedExecutions? Should it be handled here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it is not necessary.

  • UtFailedExecution does not specify the execution producer, and we want to compute priorities based on the subsystem that produced the execution. The default 'lowest' priority seems OK in that case.
  • UtFailedExecution does not contain coverage anyway, so the minimizer will handle it in a special way.

Of course, this code should probably be redesigned (and the issue we want to fix should be considered) during any new UtExecution refactoring.

@dtim dtim marked this pull request as draft September 5, 2022 13:40
@dtim dtim force-pushed the dtim/843_minimizer_prefer_symbolic branch from 3801e27 to cc8956c Compare September 5, 2022 13:52
@dtim
Copy link
Collaborator Author

dtim commented Sep 5, 2022

Following the discussions with @sergeypospelov and @Markoutte.

This PR generally implements the idea of providing additional execution priorities map to minimizer to specify that some executions are more "important" or "interesting" that others with the same coverage. It may be useful per se (although probably requires some more design), but it seems to be a hack w.r.t. the original issue (#843).

  • The temporary UtExecution class hierarchy is spreading into conceptually isolated parts, such as the minimizer.
  • It seems that this solution is only partial. Let us have two executions, a UtSymbolicExecution covering 6 instructions, and a UtFuzzedExecution that covers the same 6 instructions plus 2 other instructions, so it has a strictly better coverage. Although the symbolic execution has higher priority, the minimizer will select the fuzzed execution as its coverage is better. If we remove the fuzzed execution later during parametrized code generation, we will lose both executions, while the correct behavior would be to keep the symbolic execution, even it is not as good as the fuzzed one. At the same time, we definitely want to use the fuzzed execution in the regular case (when no parametrized tests are generated, so no fuzzed executions are removed).

As a potential alternative, we could use three stages:

  1. Select only symbolic executions.
  2. Add fuzzed executions that increase coverage (optional, turned off if generating parametrized tests).
  3. Minimize the resulting collection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Prioritize Symbolic Engine executions
3 participants