Overhaul test suite

[berleant]

I believe this is already agreed on, I'm just making the issue for the sake of completeness. The main thing is that nosetests is/will be no longer supported. We need to move to either pytest or unittest.

Some additional brainstorming:

• shorten test runtime to reduce friction
• refactor tempfile stuff, perhaps into a decorator
• refactor the @skipif(no_pkg) statements
• take out non-empty .nii files in testing
• fewer empty .nii files
• replace yield assert_* statements (not all testing tools handle iterators well, e.g. the mock module)
• check code coverage weak areas, set goal?
• remove warnings, print statements

[satra]

+1 for pytest. i was able to hack a quick test on a branch a few weeks back. the main things to change are all the cases where we create directories and then delete them. pytest has a specific form for those. also pytest comes with assert, so we may have to recreate some of the utility asserts in terms of the simply assert.

+1 to remove warnings and print statements

now that we have better testing in place, we may want to rethink what the auto tests are achieving (init, metadata, etc.,.) and whether there are better ways of testing it.

also with circle move to a bids dataset for all testing and have matrix integration tests in place for each interface. for example, did the output of an interface change with versions of the package?

let's continue the discussion here and outline a good plan.

[chrisgorgo]

I would also put on the discussion removal of the "auto_" tests. Do they
serve any other purpose than hinting that the API has changed?

On Sun, Sep 18, 2016 at 7:10 PM, Satrajit Ghosh notifications@github.com
wrote:

+1 for pytest. i was able to hack a quick test on a branch a few weeks
back. the main things to change are all the cases where we create
directories and then delete them. pytest has a specific form for those.
also pytest comes with assert, so we may have to recreate some of the
utility asserts in terms of the simply assert.

+1 to remove warnings and print statements

now that we have better testing in place, we may want to rethink what the
auto tests are achieving (init, metadata, etc.,.) and whether there are
better ways of testing it.

also with circle move to a bids dataset for all testing and have matrix
integration tests in place for each interface. for example, did the output
of an interface change with versions of the package?

let's continue the discussion here and outline a good plan.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1648 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAOkp15iIUsN6JdLUoOhVsKejUwQM7n8ks5qre8EgaJpZM4KADsP
.

[mgxd]

In place of these empty .nii files, having some real data files would allow actually running the interface - a) making sure the interface successfully runs and b) expecting certain outputs (as satra said output can change with versions)

[berleant]

The question is whether we want real data files in our repo. It's convenient, but they are (potentially) big and not suited to version control. I vote for keeping "real" data integration testing in travis/circle.

This came out of a conversation I had with @oesteban -- We might also establish a rule/guideline on what tolerance we should expect for checking equality of floats (in, e.g., numpy.close, assert_almost_equal, etc.). At what point does the difference become "real", something to look into, and when is the difference attributable to rounding errors from slightly different environments or implementation changes in dependencies? Our approach right now is very ad hoc.

[satra]

@shoshber - no files directly in the repo. but a versioned repo of files.

we can use git annex to download it for testing when necessary.

[satra]

this has mostly been achieved. as we separate out niflows, we will consider more testing.