Failures in Test suite

[tillea]

Summary

When trying to build the Debian package for version 1.6.1 there are three test suite errors.

Script/Workflow details

Here is a link to the full build log in Debian CI

It says

=========================== short test summary info ============================
FAILED ../nipype/algorithms/tests/test_CompCor.py::TestCompCor::test_compcor
FAILED ../nipype/algorithms/tests/test_CompCor.py::TestCompCor::test_compcor_variance_threshold_and_metadata
FAILED ../nipype/algorithms/tests/test_CompCor.py::TestCompCor::test_tcompcor
= 3 failed, 2674 passed, 239 skipped, 1 deselected, 5 xfailed, 25 warnings in 142.68s (0:02:22) =

Platform details:

Debian unstable. The command that is suggested here does not really help since the package was not successfully built so it gives no information about the system. Please see on top of above build log.

Kind regards, Andreas.

[effigies]

Hmm. My guess would be that there's been a change in numpy or scipy's SVD implementation. Or possibly that sign flips can happen on different instruction sets. Will see if I can reproduce locally.

[effigies]

Locally not seeing this issue. I also can't see in those logs the installation of numpy and scipy, which seem like the most likely sources of variation. Do you have numpy and scipy packages that can be installed in a virtual environment? Preferably wheels, but .debs are fine too.

[tillea]

My local build log has

python3-numpy (1:1.19.5-1)
python3-scipy (1.7.1-1)

There is also a complete build log featuring this information.

[effigies]

Yes, I can reproduce in Docker. If I install python3-numpy and python3-scipy Debian packages, then the tests fail. If I pip install --force numpy=1.19.5 to get the numpy-distributed wheel, this fixes the tests.

Would it be possible to try rebuilding the numpy package to see if there's a stochastic bug here?

[tillea]

I tried rebuilding the numpy source package on my side but went into a build error (no idea why). So I need to admit that this is harder than expected (at least with the current state of the Debian archive).

[effigies]

Thanks for checking back in. My feeling is that I would rather not make our tests accept sign flips out of SVD, as reproducibility of results is highly desirable.

If we can reliably produce a numpy build that induces these sign flips, it would be a good issue to open with numpy.

Does this seem reasonable? Let me know if there's anything I can do to help.

[emollier]

Hello,

The test failure in nipype eventually lead to Debian Bug #997126, in which Graham Inggs noticed that the failure is similar to the Debian Bug #994457 affecting r-cran-gnm. This could eventually be related to a change of SVD implementation in lapack 3.10, and indeed numpy depends on lapack, among other things.

In hope this helps...

[effigies]

Thanks for the references. It seems that we should be tolerant to sign flips of entire columns. Opened #3405. Feel free to use the patch in Debian for existing releases of nipype.

[emollier]

Thanks very much! :)