Skip to content

Coverage depending on the order of tests run #296

Closed
@srprash

Description

@srprash

While doing this PR, the coverage dropped significantly. I believe the reason for such a massive drop in the coverage is due to the fact that every test is run with coverage run py.test ... with different set of test cases. For each run of such command, only a single .coverage file is produced and overwriten by the run of next test cases.
Depending on which set of test cases are run as the command in the very last, the .coverage file (and thus the generated coverage.xml) will contain the coverage report about that run. In this case, I believe the last command to overwrite the .coverage was coverage run --source aws_xray_sdk -m py.test tests/ext/sqlalchemy_core/ --ignore tests/ext/sqlalchemy_core/test_sqlalchemy_core_2.py and so other modules' coverage dropped to 0% as per this report.

This makes the coverage report quite unreliable, and the solution is to generate different coverage report for each test run and aggregate them together in a codecov report.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions