Speed up Time for Coverage Combine

[tanmayzengit]

Is your feature request related to a problem? Please describe.
I am trying to setup a code coverage representation for my pull requests for specific number of tests. Using coverage combine to combine all the data files takes a lot of time which is not very ideal. Is there a quick resolution for this. I tried moving to 5.0 but the combine performance degraded from previous versions which has pickled dictionary as the datastore instead of sqlite.

Describe the solution you'd like
My idea is to somehow bring down the combine step in multiprocessing cases where we parallely run the tests and combine them at a later stage

Describe alternatives you've considered
For doing a per PR coverage i did consider using collector itself to have the data but i would want to avoid any usage of database from my side and if possible i would prefer using that coverage data file as that is much more convenient to debug
Additional context
Add any other context about the feature request here.

[nedbat]

Can you provide some more details? Ideally, you'd link us to your repo so that we can see the behavior you are seeing. But even if you can't do that: how long is combine taking? How many data files do you have, and how large are they? Can you show us the output of coverage debug data before the combine step?

[tanmayzengit]

Linking the repo would not be possible but, i can adda little context here.
I have updated coverage version from 4.0a6 -> 5.0.0 and was expecting an improvement in combine time.
Combine time before-> ~45 mins
Combine time after -> ~95 mins
The data files are around ~8500 with parallel and multiprocessing configurations which gives the total data.
coverage debug data gives nothing because we dont have a .coverage file before combine step?

p.s My major goal is to report coverage on a PR so if combine cannot be improved is there a better way to do this?

[nedbat]

Why are you only updating to 5.0.0? Can you try 6.5.0?
You say you have 8500 data files? Are you running your test suite 8500 times? Maybe I'm misunderstanding something.

[tanmayzengit]

Eventual goal is to move to the latest version but currently need python 2.7 support thats why 5.0
So by data file i dont mean .coverage final data file, For every subprocess i am generating a separate coverage file (as i said multiprocessing and parallel flag check here) and combining them gives me .coverage file which has the data for the entire repo.

[nedbat]

Please try 5.5 (pip install coverage==5.5). It's the latest version that runs in 2.7.

[nedbat]

Two more ideas for you:

1. Instead of using parallel, try using append to put more data into each file. You have to coordinate to be sure that simultaneous runs (for example with pytest-xdist) don't try to use the same file.

2. I wonder how much unique data is in your data files? I used a small program to assess the uniqueness of my files, and found that 1111 files had 1028 unique fingerprints. That's not a huge savings, but you might have different results. If so, hashing the data files before combining could remove files that aren't adding anything new.

import collections
import glob
import hashlib

files = list(glob.glob(".coverage.*"))
print(f"{len(files)} files")

hashes = collections.defaultdict(list)
for file in files:
    with open(file, "rb") as f:
        h = hashlib.new("sha256")
        h.update(f.read())
        sha = h.hexdigest()
        hashes[sha].append(file)

print(f"{len(hashes)} hashes")
for sha, files in hashes.items():
    if len(files) > 1:
        print(sha)
        for file in files:
            print(f"    {file}")

[nedbat]

Oh, coverage.py writes a timestamp into the data file. If I remove that, then my 1111 files have only 761 unique hashes.

[tanmayzengit]

So i found a similar issue https://github.com/nedbat/coveragepy/pull/765/files
I could not acheive any better results though when i moved to version 5.5

Instead of using parallel, try using append to put more data into each file. You have to coordinate to be sure that simultaneous runs (for example with pytest-xdist) don't try to use the same file.

Not sure if this works as expected with multiprocessing though. So i my case the parent process creates a .coverage file and all the child process keep rewriting that file so i was having issues with it. switching to parallel helped solve the issue. I will try with append though but is the behaviour same with multiprocessing set as concurrency.

Oh, coverage.py writes a timestamp into the data file. If I remove that, then my 1111 files have only 761 unique hashes.

How do we do that?