measuring coverage with multiprocessing.process

[flixman]

Describe the bug
I have a "regular" unittest script, in which I am creating an "echo server" using multitprocessing.process. To start the server I do the following:

def tcp_server(host, port):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        s.bind((host, port))
        s.listen()
        while True:
            conn, addr = s.accept()
            with conn:
                while data := conn.recv(1024):
                    conn.sendall(data)

class TestEcho(TestCase):

    def setUp(self) -> None:
        self.server = multiprocessing.Process(target=tcp_server, args=('127.0.0.1', 1235))
        self.server.start()

    def tearDown(self) -> None:
        self.server.terminate()

        while self.server.is_alive()
            sleep(0.1)

    def test1(self):
       .....

my .coveragerc looks like this:

[run]
parallel = True
concurrency = multiprocessing

[report]
omit = tests/**/__init__.py

To run my tests, I use the following script:

export COVERAGE_PROCESS_START=.coveragerc
coverage run -m unittest discover -s ./tests
coverage combine
coverage report -m

and when getting the report... there are files that I am sure have been executed (as the tests are successful) but they get reported as missing.

To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

1. What version of Python are you using? 3.10.1
2. What version of coverage.py shows the problem? 6.2
3. What versions of what packages do you have installed?

asgiref==3.4.1
blinker==1.4
Brotli==1.0.9
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.9
click==8.0.3
colorama==0.4.4
container-app-conf==5.2.2
coverage==6.2
cryptography==3.4.8
cycler==0.11.0
ffmpeg-normalize==1.22.4
ffmpeg-progress-yield==0.2.0
Flask==2.0.2
Flask-SQLAlchemy==2.5.1
fonttools==4.28.5
forex-python==1.8
greenlet==1.1.2
h11==0.12.0
h2==4.1.0
hpack==4.0.0
hyperframe==6.0.1
idna==3.3
itsdangerous==2.0.1
Jinja2==3.0.3
kaitaistruct==0.9
kiwisolver==1.3.2
ldap3==2.9.1
MarkupSafe==2.0.1
matplotlib==3.5.1
mitmproxy==7.0.4
msgpack==1.0.3
numpy==1.21.5
packaging==21.3
pandas==1.3.5
passlib==1.7.4
pdftotext==2.2.2
Pillow==8.4.0
pip-date==1.0.3
protobuf==3.18.1
publicsuffix2==2.20191221
py-range-parse==1.0.5
pyasn1==0.4.8
pycparser==2.21
pycryptodome==3.12.0
pyOpenSSL==20.0.1
pyparsing==2.4.7
pyperclip==1.8.2
pyserial==3.5
python-dateutil==2.8.2
python-dotenv==0.19.2
pytimeparse==1.1.8
pytz==2021.3
PyYAML==6.0
requests==2.26.0
ruamel.yaml==0.17.16
ruamel.yaml.clib==0.2.6
scipy==1.7.3
simplejson==3.17.6
six==1.16.0
sortedcontainers==2.4.0
SQLAlchemy==1.4.29
tenacity==8.0.1
toml==0.10.2
tornado==6.1
tqdm==4.62.3
urllib3==1.26.7
urwid==2.1.2
voluptuous==0.12.2
Werkzeug==2.0.2
wsproto==1.0.0
XlsxWriter==3.0.2
youtube-dl==2021.12.17
zstandard==0.15.2

Additional context
Add any other context about the problem here.

[nedbat]

Hi, you didn't answer these questions from the template:

4. What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix.
5. What commands did you run?

Did you create a .pth file as described in https://coverage.readthedocs.io/en/6.2/subprocess.html ?

[flixman]

Hey @nedbat, sorry for missing those:

4. In this gist there is a test that is showing this behavior.
5. To run that, assumming I have it in a folder called 'test_coverage' in the root of my project:

export COVERAGE_PROCESS_START=.coveragerc
coverage run -m unittest discover -s ./test_coverage
coverage combine
coverage report -m

The output I am getting is:

.
----------------------------------------------------------------------
Ran 1 test in 1.104s

OK
Combined data file .coverage.polaris.86440.668865
Name                            Stmts   Miss  Cover   Missing
-------------------------------------------------------------
test_coverage/test_example.py      37      9    76%   8-16
-------------------------------------------------------------
TOTAL                              37      9    76%

Answering your final question: Yes: I am using python from a virtual environment, and I have created a file named venv/lib/python3.10/site-packages/coverage.pth containing

import coverage; coverage.process_startup()

So.. what am I doing wrong? Thank you very much for your help!

[nedbat]

I think the problem here is self.server.terminate(). terminate doesn't clean up the process cleanly, so it doesn't have a chance to write its coverage data. You'll somehow need to arrange for tcp_server to return, and avoid using terminate(). I don't know the best way: a shared value that you can trigger from tearDown? That would require making the sockets non-blocking, I think, but I'm no expert. Or perhaps you could send a final message on the socket that the server would recognize and return.

I'm a little surprised this hasn't come up before. I'll add a note to the docs.

[nedbat]

On second thought, I want to explore whether I can write out data on the terminate signal.

[nedbat]

I can't run your complete test case (s.connect((host, port)) causes Invalid Argument), but it runs enough to show the problem, and I think I have a branch of coverage.py that fixes it. Can you try this out?

pip install git+https://github.com/nedbat/coveragepy.git@nedbat/fix-1307