Add tests for fuzzing platform #1516
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Markoutte
requested review from
denis-fokin,
Vassiliy-Kudryashov and
volivan239
Vassiliy-Kudryashov
approved these changes
Dec 14, 2022
volivan239
approved these changes
Dec 14, 2022
Comment on lines
+60
to
+86
| Assertions.assertEquals(1, p1.generate(description, Unit).count()) | ||
| Assertions.assertEquals(1, (p1.generate(description, Unit).first() as Seed.Simple).value) | ||
|
|
||
| Assertions.assertEquals(1, p2.generate(description, Unit).count()) | ||
| Assertions.assertEquals(2, (p2.generate(description, Unit).first() as Seed.Simple).value) | ||
|
|
||
| Assertions.assertEquals(1, p3.generate(description, Unit).count()) | ||
| Assertions.assertEquals(3, (p3.generate(description, Unit).first() as Seed.Simple).value) | ||
|
|
||
| Assertions.assertEquals(1, p4.generate(description, Unit).count()) | ||
| Assertions.assertEquals(4, (p4.generate(description, Unit).first() as Seed.Simple).value) | ||
|
|
||
| Assertions.assertEquals(2, m1.generate(description, Unit).count()) | ||
| Assertions.assertEquals(1, (m1.generate(description, Unit).first() as Seed.Simple).value) | ||
| Assertions.assertEquals(2, (m1.generate(description, Unit).drop(1).first() as Seed.Simple).value) | ||
|
|
||
| Assertions.assertEquals(2, m2.generate(description, Unit).count()) | ||
| Assertions.assertEquals(3, (m2.generate(description, Unit).first() as Seed.Simple).value) | ||
| Assertions.assertEquals(4, (m2.generate(description, Unit).drop(1).first() as Seed.Simple).value) | ||
|
|
||
| Assertions.assertEquals(4, m3.generate(description, Unit).count()) | ||
| Assertions.assertEquals(1, (m3.generate(description, Unit).first() as Seed.Simple).value) | ||
| Assertions.assertEquals(2, (m3.generate(description, Unit).drop(1).first() as Seed.Simple).value) | ||
| Assertions.assertEquals(3, (m3.generate(description, Unit).drop(2).first() as Seed.Simple).value) | ||
| Assertions.assertEquals(4, (m3.generate(description, Unit).drop(3).first() as Seed.Simple).value) | ||
|
|
||
| Assertions.assertEquals(4, m4.generate(description, Unit).count()) |
There was a problem hiding this comment.
Why not to import assertEquals (here and in some other files)?
Comment on lines
+16
to
+19
| @Test | ||
| fun `string generates same values`() { | ||
| fun collect(): List<String> { | ||
| return runBlockingWithContext { |
There was a problem hiding this comment.
Just thoughts: if we want reproducibility in many/most/all cases, maybe we should find a more general way to check it? E.g., introduce something like
fun <T> getReproducibleResults(generator: () -> T): T {
val probe1 = generator()
val probe2 = generator()
assertEquals(probe1, probe2)
return probe1
}Then we can wrap calls to fuzzer in tests in such function and process results as usual. (Just an example, there are lots of other ways to check reproducibility more general).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fuzzing platform should generate reproducible tests on each run with new seeded random object and be cancelable by coroutine context. This PR fixes these problems and adds tests for core functions of fuzzing platform.
Type of Change
How Has This Been Tested?
Regression and integration tests
New tests have been added:
Manual Scenario
Tests now have to be reproducible. It means, that fuzzing with more time to run has all those tests that fuzzing with less time to run has. Try to run fuzzing on
org.utbot.fuzzing.samples.Strings#test(String s)with different timeouts.Checklist: