Update time management of python test generation

[tamarinvs19]

Description

New test generation strategy is:

• If function is fully annotated and type inference algorithm cannot infer type we use all time from UI-settings to fuzzing and test generation;
• If type inference and infer type we try to fuzz tests for one annotation with the following restrictions:
  • max 150 successfully executable tests (with or without exeception)
  • max 10 invalid tests (when we generate incorrect arguments and have TypeError or AttributeError; or we cannot generate any valid representative of the inferred type)
  • and 5 additional tests (as a precaution)
  • until time limit

In future we can modify strategies and add it to UI. For example, wait max coverage.

Fixes #1874
Fixes #1831

How to test

Manual tests

Fully annotated function

For example,

def div(a: int, b: int) :
    return a / b

Expected: two tests (valid and with ZeroDivisionError), test generation process lasted all time from UI-settings

Or,

def str_test(x: str):
    if x[0] == 'b':
        if x[1] == 'a':
            if x[2] == 'd':
                return 'Very bad!'

Expected: all time generation, one test with x = 'ba' or longer (with timeout 60s) and all covered branches with timeout 180s

Without or partially annotated function

For example,

def str_test(x):
    if x[0] == 'b':
        if x[1] == 'a':
            if x[2] == 'd':
                return 'Very bad!'

Expected: all time generation, one test with x = 'b', no one with 'ba' and longer (regardless of timeout)

Self-check list

Check off the item if the statement is true. Hint: [x] is a marked item.

Please do not delete the list or its items.

• I've set the proper labels for my PR (at least, for category and component).
• PR title and description are clear and intelligible.
• I've added enough comments to my code, particularly in hard-to-understand areas.
• The functionality I've repaired, changed or added is covered with automated tests.
• Manual tests have been provided optionally.
• The documentation for the functionality I've been working on is up-to-date.

[tochilinak]

Seems like this is wrong. Example:

def f(x: list): pass

In this case we need type inference for intermal type.

We can get information about whether function signature has Any's inside type inference algorithm. In that case initial state has no Any nodes, and the main loop makes only one iteration (initial state -> no new states -> finish).

[tochilinak]

Problem with setting limitManager.mode = TimeoutMode.

[tamarinvs19]

Problem with setting limitManager.mode = TimeoutMode.

Fixed

[tochilinak]

Initially this code was needed to run fuzzing with initial types (even when those are not full). Will we run fuzzing with list[Any] for the following function?

def f(x: list): ...

[tamarinvs19]

Maybe. But only if all arguments have not-typing.Any annotations. And if all annotations are full we run fuzzing twice

[tochilinak]

Are you sure? Type inference does not run annotationHandler on initial signature.

[tamarinvs19]

Now I run fuzzing with initial annotations only after type inference) But I can change program and run fuzzing twice with initial annotations, but I think it isn't a good solution

[tamarinvs19]

I added handling of an initial annotation after first expansion in BaselineAlgorithm.

• if annotation is full we cannot expand it and stop type inference and then start fuzzing;
• if annotation is not full we try to expand and first of all try to fuzz initial annotation (for example, if it is list[typing.Any] we can generate empty list), after that we continue type inference

[Markoutte]

On line 215 (sorry, that I comment another line, because I was not able to find how to comment 215) there's no check that:

is ValidExecution -> {
  val trieNode: Trie.Node<Instruction> = description.tracer.add(coveredInstructions)

Returns the node the was already found. You can check trieNode.count > 1 to ignore duplicates and minimise the count of tests for user. At the moment similar tests are generated with different arguments, but same trace.

[tamarinvs19]

Fixed