Prefer symbolic executions during minimization #854
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Suppose we have two executions with the same coverage, one produced by the symbolic engine, and one produced by the fuzzer. From the user's point of view, they are equivalent, and the minimizer can keep any of them. Usually it will be fuzzed one, because fuzzer produces executions earlier that the symbolic engine, and the minimizer orders executions with the same coverage by their order in the execution list.
When parametrized tests are generated, fuzzed tests tend to be excluded due to mock processing issues. If a fuzzed tests has been selected by the minimizer instead of the symbolic one, the corresponding path remains uncovered: one (symbolic) test has been dropped by the minimizer, and another (fuzzed) test has been excluded by the code generator.
The coverage of the entire parametrized test suite may be improved is minimizer selects symbolic executions when possible. It does not guarantee the full coverage (it is possible to have only fuzzed tests for some paths), but we expect that the coverage may be improved that way.
To let users to control the minimizer behavior, new
UtSettings
option is introduced:preferSymbolicExecutionsDuringMinimization
. When it is set totrue
(default), the minimizer will consider the execution source (specifically, its class:UtSymbolicExecution
orUtFuzzedExecution
(UtFailedExecution
and any potential other directUtExecution
subclasses are assigned the lowest priority, butUtFailedExecution
does not have any coverage anyway, so it is processed separately).Fixes #843
Type of Change
This fix changes the set of generated tests. It should not decrease coverage, and is expected to increase coverage when generating parametrized tests (please report a bug if coverage is decreased when
preferSymbolicExecutionsDuringMinimization
istrue
).This PR also changes the minimizer interface.
How Has This Been Tested?
Automated Testing
All existing minimizer tests should pass.
Two new minimizer tests have been added:
org.utbot.framework.minimization.MinimizationGreedyEssentialTest#testWithSourcePriority
org.utbot.framework.minimization.MinimizationGreedyEssentialTest#testWithoutSourcePriority
Manual Scenario
Make sure that fuzzer is enabled in
UtSettings
, and both fuzzer and symbolic engine are enabled in the plugin configuration window.Create any method simple enough so both fuzzer and symbolic engine can cover all branches. An example:
Set
UtSettings.preferSymbolicExecutionsDuringMinimization
totrue
and generate a test suite forfoo
. Test cases produced by the symbolic engine should be generated.Set
UtSettings.preferSymbolicExecutionsDuringMinimization
tofalse
and generate a test suite forfoo
again. This time, the plugin should generate test cases produced by the fuzzer.Checklist (remove irrelevant options):
This is the author self-check list