Mutate values to improve code coverage #213 #713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

Markoutte merged 4 commits into main from pelevin/213_Mutate_values_to_improve_code_coverage

Aug 24, 2022

Collaborator

Markoutte commented Aug 11, 2022

Description

This commit adds a mutation mechanism to the fuzzing for improving code coverage by applying random mutations for numbers and strings.

Fixes #213

Type of Change

Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

Manual Scenario

Turn on only fuzzing mode. These examples can find (usually do for default utbot settings):

Should find the string that starts with prefix "bad!"

public static void test(String s) {
    if (s.charAt(0) == "b".charAt(0)) {
        if (s.charAt(1) == "a".charAt(0)) {
            if (s.charAt(2) == "d".charAt(0)) {
                if (s.charAt(3) == "!".charAt(0)) {
                    throw new IllegalArgumentException();
                }
            }
        }
    }
}

Should find a string that has "!" in the 5-th position, but can totally change the other part.

public static void testStrRem(String str) {
    if (!"world???".equals(str) && str.charAt(5) == '!' && str.length() == 6) {
        throw new RuntimeException();
    }
}

Should find the value that will return 2 for the method:

public int floatToInt(float x) {
    if (x < 0) {
        if ((int) x < 0) {
            return 1;
        }
        return 2; // smth small to int zero
    }
    return 3;
}

If results are failed try to increase test generation time.

Checklist:

The change followed the style guidelines of the UTBot project
Self-review of the code is passed
The change contains enough commentaries, particularly in hard-to-understand areas
New documentation is provided or existed one is altered
No new warnings
New tests have been added
All tests pass locally with my changes

Markoutte added the comp-fuzzing label

Markoutte requested review from denis-fokin and CaelmBleidd

August 11, 2022 13:57

CaelmBleidd requested changes

View reviewed changes

utbot-framework/src/main/kotlin/org/utbot/engine/UtBotSymbolicEngine.kt Outdated

@@ @@ -436,7 +439,34 @@ class UtBotSymbolicEngine( @@
                               limitValuesCreatedByFieldAccessors = 500
                           })
                       }
-                      fuzzedValues.forEach { values ->
+                      val frequencyRandom = FrequencyRandom(Random(221))

Member

CaelmBleidd Aug 12, 2022

Shouldn't it be a configurable constant?

Collaborator Author

Markoutte Aug 16, 2022

It shouldn't because its a random seed for a Random. No difference to fuzzing if value is changed.

utbot-framework/src/main/kotlin/org/utbot/engine/UtBotSymbolicEngine.kt Outdated

-                      fuzzedValues.forEach { values ->
+                      val frequencyRandom = FrequencyRandom(Random(221))
+                      val factor = maxOf(0, UtSettings.fuzzingRandomMutationsFactor)

Member

CaelmBleidd Aug 12, 2022

Personally, I'd prefer to see coerceAtLeast function here since it shows the semantics better

Collaborator Author

Markoutte Aug 16, 2022

For me it hides the value when variable name is long:

var onlyPositiveValue = theRandomValuedRetrievedFromFile.coearceAtLeast(0)

and

var onlyPositiveValue = maxOf(0, theRandomValuedRetrievedFromFile)

But I don't have any preferable way here.

utbot-framework/src/main/kotlin/org/utbot/engine/UtBotSymbolicEngine.kt Outdated

+                          val fvi = fuzzedValues.iterator()
+                          while (fvi.hasNext()) {
+                              yield(fvi.next())
+                              // if we stuck switch to "next + several mutations" mode

Member

CaelmBleidd Aug 12, 2022

Please, specify what stuck means and how you detect it. Is it connected with (attempts + 1) % (factor / 100) == 0 below? It's not an obvious condition for me as well

utbot-framework/src/main/kotlin/org/utbot/engine/UtBotSymbolicEngine.kt Outdated

+                          while (fvi.hasNext()) {
+                              yield(fvi.next())
+                              // if we stuck switch to "next + several mutations" mode
+                              if ((attempts + 1) % (factor / 100) == 0 && coveredInstructionValues.isNotEmpty()) {

Member

CaelmBleidd Aug 12, 2022

factor / 100 can be zero, can`t it?

utbot-framework/src/main/kotlin/org/utbot/engine/UtBotSymbolicEngine.kt Outdated

                                       return@flow
                                   }
+                                  // Update the seeded values sometimes
+                                  // This is necessary because some values cannot do a good values in mutation in any case
+                                  if (frequencyRandom.random.nextInt(1, 101) <= 50) {

Member

CaelmBleidd Aug 12, 2022

Isn't is just nextBoolean()?

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/mutators/StringRandomMutator.kt Outdated

+                      return buildString {
+                          append(value.substring(0, position))
+                          if (random.nextBoolean()) {
+                              append(charToMutate - random.nextInt(1, 128))

Member

CaelmBleidd Aug 12, 2022

What 128 means here?

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/mutators/StringRandomMutator.kt Outdated

+                  }
+                  private fun tryRemoveChar(random: Random, value: String, position: Int): String? {
+                      if (value.isEmpty() || position >= value.length) return null

Member

CaelmBleidd Aug 12, 2022

could be just position >= value.length

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/Fuzzer.kt Outdated

                       PrimitiveWrapperModelProvider,
                   )
               }
+              fun defaultMutators(): List<ModelMutator> = listOf(StringRandomMutator(50), NumberRandomMutator(50))

Member

CaelmBleidd Aug 12, 2022

constants without names

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/ModelMutator.kt Outdated

+                      parameters: List<FuzzedValue>,
+                      random: Random,
+                  ) : List<FuzzedValue> {
+                      return parameters.mapIndexed { index, fuzzedValue ->

Member

CaelmBleidd Aug 12, 2022

here and in other functions expression body might be used

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/ModelMutator.kt Outdated

+                      value: FuzzedValue,
+                      random: Random
+                  ) : FuzzedValue? {
+                      return if (random.nextInt(1, 101) < probability) value else null

Member

CaelmBleidd Aug 12, 2022

random.nextInt(1, 101) < probability used in many places. May you create a function for it?

Markoutte added 3 commits

August 22, 2022 09:54


          Mutate values to improve code coverage #213

5fd4d6e


          Fix tests because StringConstantModelProvider doesn't mutate value an…

71aac5e

…ymore


          Refactoring fuzzing mutations

a2aa85c

Markoutte force-pushed the pelevin/213_Mutate_values_to_improve_code_coverage branch from 48d6e4b to a2aa85c Compare

August 22, 2022 10:50

Markoutte requested a review from CaelmBleidd

August 22, 2022 13:21

CaelmBleidd approved these changes

View reviewed changes

Member

CaelmBleidd left a comment

overall lgtm

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/Fuzzer.kt Outdated

+               * Wraps sequence of values, iterates through them and mutates.
+               *
+               * Mutation when possible is generated after every value of source sequence and then supplies values until it needed.
+               * [statistics] should is not updated by this method, but can be changed by caller.

Member

CaelmBleidd Aug 22, 2022

Should not be updated?

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/Fuzzer.kt Outdated

+                      yield(fvi.next())
+                      // Fuzzing can generate values that doesn't recover new paths.
+                      // So, fuzzing tries to mutate values on each loop
+                      // if there are too much attempts to find new paths without mutations.

Member

CaelmBleidd Aug 22, 2022

that don't recover and too many

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/Fuzzer.kt Outdated

+                              newValues[index] = value
+                          }
+                      }
+                  return newValues.takeIf { it != values }

Member

CaelmBleidd Aug 22, 2022

How many values do we have here? Wouldn't it be too expensive to check the equality of the two lists? Maybe we can store some flag that a mutation has happened?

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/ModelMutator.kt Outdated

Comment on lines 23 to 28

+                          .mapIndexed { index, fuzzedValue ->
+                              mutate(description, index, fuzzedValue, random)?.let { mutated ->
+                                  FuzzedParameter(index, mutated)
+                              }
+                          }
+                          .filterNotNull()

Member

CaelmBleidd Aug 22, 2022

mapIndexedNotNull

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/FuzzerStatistcs.kt Outdated

+              /**
+               * Stores information that can be useful for fuzzing such as coverage, run count, etc.
+               */
+              interface FuzzerStatistics<K> {

Member

CaelmBleidd Aug 22, 2022

Misspell in the file name

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/RandomExtensions.kt

+               * @return the index of the chosen item.
+               */
+              fun Random.chooseOne(frequencies: DoubleArray): Int {
+                  val total = frequencies.sum()

Member

CaelmBleidd Aug 22, 2022

Perhaps, is frequencies could be a big array, we can create a custom collection for such frequencies that knowns information about its elements, such as total sum, statistics and other

utbot-fuzzers/src/main/kotlin/org/utbot/fuzzer/mutators/StringRandomMutator.kt Outdated

+                  private fun tryAddChar(random: Random, value: String, position: Int): String {
+                      val charToMutate = if (value.isNotEmpty()) {
+                          value[random.nextInt(value.length)]

Member

CaelmBleidd Aug 22, 2022

value.random(random)?


          Small changes after review

2716a86

Markoutte merged commit 1c44797 into main

Markoutte deleted the pelevin/213_Mutate_values_to_improve_code_coverage branch

August 24, 2022 06:44

Markoutte mentioned this pull request

Correct test descriptions for fuzzing #913

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels