Add support for editable wheels

[FFY00]

The last thing preventing this from being adopted in the wild is probably the missing support for editable wheels.

For the implementation, the idea is simple, create an editable wheel with the files from the build directory. The editable install would always need to be tied down to the build directory, which is the most annoying thing with this approach, but doing it other way would be more difficult, and very likely require some more work from the build system users.

The main worries with the implementation is that we might need to fix up some RPATHs, but I am hoping we won't, as that would bring in more complexity.
We will also likely need to do some code refactoring to re-use the logic from the normal wheel building, but I suspect that might end being too complex and not worth it. In which case, we might be better off with a little bit of code duplication. The jury's still out on that, we need to investigate and then make a call.

@rgommers does this sound reasonable to you?

[rgommers]

The editable install would always need to be tied down to the build directory

I'm not sure what this means exactly, can you elaborate?

The files needed for a working package are split between in-tree and build-dir. My understanding was that you planned to write out an install directory with symlinks for all files, with those symlinks pointing back to both directories.

[FFY00]

Not really, if I have an install directory, we'd have to run meson install to use it. That can be problematic as it is very tricky for us to inject code to do so on import. The proper distributable wheel is derived from a meson install installation. What I am planning to do here is to use the build directory itself instead.

Does that make sense to you? Is there anything you want me to clarify?

[rgommers]

Ah, do you mean put symlinks into the build dir, pointing back at .py files (and data files etc.) that are under version control in-tree? If so, that makes sense, yes.

[FFY00]

No, the editable wheel would include symlinks to the build directory or source tree, depending if the file is generated or not.

The current approach for normal wheels is to run meson install and get the files from there, here we would be creating a wheel with symlinks to the build directory or source tree directly, intead of going through meson install.

The issue is that we would depend on the build directory existing, but I don't think it is that big of a deal since we already depend on the source tree anyway.

An alternative would be to maintain our own build directory, but that has a couple issues which I can go into more detail if you want.

[rgommers]

No, the editable wheel would include symlinks to the build directory or source tree, depending if the file is generated or not.

Ah, now I get it. I thought there was an issue with this, that's why I didn't make the connection that the symlinks would be in the wheel. Found the discussion now: cupy/cupy#6261 (comment). It says " when generating a wheel all symlinks are "materialized"". Not sure if that applies here or not - I guess not, because the root cause is the wheel package not providing support for symlinks (the CuPy issue links to pypa/wheel#203), and we're not using wheel here.

What I had expected was writing directly a separate directory of symlinks, and then use a .pth file or another mechanism to point to that directory.

Okay, sounds like this should work and if it does, seems like a good plan to me.

The issue is that we would depend on the build directory existing, but I don't think it is that big of a deal since we already depend on the source tree anyway.

Agreed, that's perfectly fine I'd say.

[FFY00]

What I had expected was writing directly a separate directory of symlinks, and then use a .pth file or another mechanism to point to that directory.

That would be a bit tricky, and is not really needed since we can include symlinks in the wheel.

I think long term my plan would probably be to add an import hook, which would do the same job as a .pth file but make the implementation easier and more efficient. Unfortunately, there isn't a mechanism to inject the hook yet, and that is something I want to work on. I think there is an agreement already on how this should work (I couldn't find the links from a quick search), so we mostly just need to drive the PEP.

[rgommers]

I think long term my plan would probably be to add an import hook

Now I'm a little confused again - if the wheel contains all the symlinks we need, why do we also need an import hook? Or are these two unrelated points?

[FFY00]

That is just a future plan, which would be similar to what you described initially with the .pth file. Right now the simplest approach is just to put symlinks in the wheel, later we might want to implement some checks and other stuff to make the user experience better, but that requires running code on import, which will require an approach like that 😅

[leofang]

Right now the simplest approach is just to put symlinks in the wheel

I am confused. The madness about wheel support is you can't do this.

[FFY00]

Yeah, I was looking into that. The issue is that pypa/wheel does not support extracting symlinks, making this approach unusable, as most projects use it to extract wheels.

We will have to go with the more complex approach of using a .pth file.