pip editable install for user broken since adding pyproject.toml

[matthew-brett]

Before commit c032138, that added pyproject.toml, I could do this:

pip install --user -e .

After that commit, I get:

    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help
    
    error: option --user not recognized

I'm not quite sure why. Any ideas?

[effigies]

Ugh. Looks like this is a known issue: pypa/pip#6375

The solution seems to be --no-use-pep517, added in pypa/pip#6370.

I'm looking into whether pyproject.toml can be fixed to allow pip to handle this case without a special flag.

[effigies]

Might just drop pyproject.toml. I was hoping that it would avoid issues with an old setuptools, but it seems like a rat's nest.

[effigies]

To dig into it a bit more, I think I (sort of) understand what's going on.

	--editable	--editable
--user	build wheel, install in environment	Call setup.py develop
--user	build wheel, install in user directory	XXXX

• pyproject.toml enables build isolation, which we were using to ensure that a recent setuptools got used without depending on users to have a very recent setuptools and pip.
• pip will handle pip install --user . by building the wheel in isolation, and then installing the wheel according to --user.
• Pip knows enough about -e to know that it's setuptools' problem, so passes everything that way. Setuptools, of course, doesn't know anything about --user.

More broadly, PEP-517 simply doesn't address the notion of editable mode, which is a setuptools-specific concept. No other build systems have an equivalent.

The only resolution that currently exists to do both --user and --editable is to disable PEP-517, which can currently be done in pip with --no-use-pep517 or by deleting pyproject.toml.

[effigies]

@matthew-brett Do you feel we should remove pyproject.toml, or are you content to use --no-use-pep517 in this context?

I'm honestly not happy with any options at this point, so can be persuaded to go either way.

[matthew-brett]

I'm fine either way - but - what is the advantage of pyproject.toml?

[effigies]

Build dependencies get installed in a clean environment for the build, so old setuptools or missing wheel don't produce inscrutable errors.

[matthew-brett]

Is this problem in practice? Are there systems out there, that are installing without wheel? Can we avoid the setuptools problem by releasing wheels as well as sdists?

[effigies]

I don't know that it is a problem in practice. At the time it seemed like a cheap win that would prevent a particular class of bug report. But without a build dependency like cython where the build environment is trickier to get right, it may be overkill.

And we do release wheels.

[matthew-brett]

OK - so then, with wheels, we've covered everyone using pip.

The remaining user must be someone trying to install nibabel from an sdist or source checkout, using python setup.py install or similar - is that right? And from that subset, we are trying to help the further subset who have an outdated setuptools or no installation of wheel. I think this must be a very small group, and presumably they will tend to be more expert than average.

[effigies]

Fair enough. I'm okay dropping pyproject.toml for now.

[effigies]

Done in maint/3.0.x. Will be in master on Monday.