Skip to content

Flaky behavior of the tests depending on visualization option #598

Closed
@CaelmBleidd

Description

@CaelmBleidd

Description

Several tests for wrappers fail (or pass) the pipeline depending on the visualisation option value in UtSettings file. If we have this option enabled, everything works fine. If we disable it -- several tests fail.

To Reproduce

Steps to reproduce the behaviour:

It is flaky behaviour and there is no common way to reproduce it. Probably, it depends on the state of the system.

Visual proofs (screenshots, logs, images)

There are no specific visual proofs.
For example, org.utbot.examples.collections.MapEntrySetTest#testIteratorNext fails with absent second branch that leads to the NoSuchElementException.

Environment

No specific environment is required.

Additional context

I've investigated the problem recently. It happens because of the non-deterministic behaviour of the method peekImpl in NonUniformRandomSearch method.

When we have enabled visualization, it calls peek several times before each poll call to visualize the graph, which changes the next chosen state (because of the random usage in the method). At the same time, if we disable visualization, there will be no peek calls, and, therefore, we might have a different sequence of the chosen states from the execution queue.

Metadata

Metadata

Assignees

Labels

comp-symbolic-engineIssue is related to the symbolic execution enginectg-bugIssue is a bug

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions