Coverage in parallel failing

[rowleya]

Describe the bug
When we run pytest using multiple processes with coverage on our tests, we get an error message that seems to be related to the integration of SQLite. The error we get is available from our integration test suite here:
http://apollo.cs.man.ac.uk:8080/blue/organizations/jenkins/sPyNNaker8%20Integration%20Tests/detail/neuron_recorder_agdr/10/pipeline/102

Replicated a sample of the output here in case this disappears!

[gw7] linux -- Python 3.6.7 /usr/bin/python3
self = <SqliteDb @0x7f9efba0c1d0 debug=None filename='/var/lib/jenkins/workspace/ation_Tests_neuron_recorder_agdr/.coverage' nest=0 con=None>
sql = 'insert into meta (key, value) values (?, ?)'
parameters = ('has_arcs', '1')

    def execute(self, sql, parameters=()):
        """Same as :meth:`python:sqlite3.Connection.execute`."""
        if self.debug:
            tail = " with {!r}".format(parameters) if parameters else ""
            self.debug.write("Executing {!r}{}".format(sql, tail))
        try:
>           return self.con.execute(sql, parameters)
E           sqlite3.IntegrityError: UNIQUE constraint failed: meta.key

To Reproduce
How can we reproduce the problem? Please be specific.

1. What version of Python are you using?
   3.6
2. What version of coverage.py are you using? The output of coverage debug sys is helpful.
   5.0
3. What versions of what packages do you have installed? The output of pip freeze is helpful.
   See above link for details.
4. What code are you running? Give us a specific commit of a specific repo that we can check out.
   Integration tests in Jenkins; see above for details.
5. What commands did you run?

py.test SpiNNUtils/unittests -rs -n auto --forked --show-progress --cov-branch --cov spynnaker8 --cov spynnaker --cov spinn_front_end_common --cov pacman --cov data_specification --cov spinnman --cov spinn_machine --cov spinn_storage_handlers --cov spalloc --cov spinn_utilities --junitxml junit/SpiNNUtils.xml --cov-report xml:coverage.xml --cov-append --timeout 1200

Expected behavior
A clear and concise description of what you expected to happen.
No error message; protection against writing from mulitple processes through checking for key in database before attempted insert.

Additional context
Add any other context about the problem here.

[kstaniek]

Facing the same issue. Is there any workaround, pls?

[nedbat]

The simplest way to avoid this issue for now is to pin coverage.py to 4.5.4. Another thing that would help me find a solution for you is to provide me with specific, complete instructions that would let me reproduce the problem.

[nedbat]

@rowleya Can you give me specific instructions I can follow to reproduce the problem? What repo, what commit, what commands to run?

[AlexandreYang]

I couldn't reproduce reliably the issue we have on this CI Pipeline https://dev.azure.com/datadoghq/integrations-core/_build/results?buildId=4968&view=logs&jobId=6856723d-645b-5a7b-98a0-8af874afe33f&j=82f604ee-2edd-5639-6463-dbdc7756af1f&t=1d9be0b0-41b5-5871-19d1-c01bf70ecbd1

But here are few things that might help:

• The bug happen when using pytest with both --cov-append --cov=X options. Without --cov-append, it works fine.
• When the error sqlite3.IntegrityError: UNIQUE constraint failed: meta.key is raised (while inserting a new entry with has_arcs key in meta table), there is already an entry in the meta table with ('has_arcs', '0')

[rowleya]

The specific instructions that seem to fail for me are:

python3.6 -m virtualenv test
cd test
source bin/activate
git clone https://github.com/SpiNNakerManchester/SpiNNUtils.git
cd SpiNNUtils && python setup.py develop && cd ..
pip install python-coveralls "coverage>=4.4"
pip install pytest-instafail pytest-xdist pytest-cov pytest-progress pytest-timeout pytest-forked
py.test SpiNNUtils/unittests -rs -n auto --forked --show-progress --cov-branch --cov spynnaker8 --cov spynnaker --cov spinn_front_end_common --cov pacman --cov data_specification --cov spinnman --cov spinn_machine --cov spinn_storage_handlers --cov spalloc --cov spinn_utilities --junitxml junit/SpiNNUtils.xml --cov-report xml:coverage.xml --cov-append --timeout 1200

[rowleya]

It makes sense that --cov-append is the issue; I am trying to achieve something specific with this, which is to run lots of tests from different repositories, including integration tests, and get the overall coverage, so this is essential.

A quick test shows that without the -n auto --forked but with the other arguments (including --cov-append) this also works. It fails some of the tests due to the above not installing all the requirements, but the coverage module doesn't fail; the extra dependencies are:

pip install numpy testfixtures httpretty