Restore shared extension module and/or dependency libraries

[indygreg]

This project was created in support of PyOxidizer. And a goal of PyOxidizer was to see if it was possible to achieve a single file executable Python application.

In order to achieve that goal, this project built highly statically linked Python distributions.

On Linux and macOS, extension modules are statically linked into libpython. And 3rd party library dependencies of extension modules (OpenSSL, SQLite, etc) are also statically linked into libpython.

This mostly just works. Until it doesn't.

For example, ELF symbols resolution prefers symbols in the global namespace over a local namespace. So if you load a .so providing symbol foo which is also provided by an external libpython.so, the libpython.so symbol is used. This leads to wonkiness like https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html#static-linking-of-libx11-incompatibility-with-pyqt-on-linux.

As the Python distributions in this project have grown a life outside of PyOxidizer, the opinionated choice of statically linking all extension modules and shared libraries is hard to justify. We're foregoing compatibility with aspects of the larger Python ecosystem.

So I'm proposing that we revert to distributing 3rd party libraries and extension modules as shared libraries.

[indygreg]

I looked at this over the weekend. I was soft blocked after running into a bug with patchelf corrupting ELF binaries. I need to read up on how the GNU dynamic loader handles rpath and $ORIGIN before I continue with an alternate solution.

[mzpqnxow]

Just stumbled across this project and wish I had found it a year ago

I have a (much, much) less ambitious "project" (a couple of scripts really) here that may be helpful as a reference for $ORIGIN linking

My use-case is probably unusual - I need to have Python linked against a few older as well as custom-patched OpenSSL versions. I didn't want to deal with setting runtime linking environment variables (e.g. LD_LIBRARY_PATH) and didn't want to pollute my system which is why I ended up with $ORIGIN. It's not exactly what this project aims to help with, but there's overlap in the requirements, obviously

The junk I have at that repo is not something that you'll be able to copy/paste but I hope it might be a little helpful - even if just to see what flags to use and how the syntax looks - because there's a $ in $ORIGIN, and because the string is actually passed as a command there's some wacky double-escaping

You'll be much better off if you stick with Python >= 3.10. If I remember correctly, on <= 3.8/3.9 there are some annoyances that manifest as a a silently skipped _ssl.so module , due to a superficial build-time test

[mzpqnxow]

There's not much there, but here's the bit you'll want if you decide to take this approach

All you probably care about are the -Wl flags...

OPENSSL_V=1.1.1k
OPENSSL_RPATH="../../openssl-$OPENSSL_V/lib"
PY_RPATH="../lib"
LDFLAGS="-Wl,-rpath,'\$\$ORIGIN/$PY_RPATH' -Wl,-rpath,'\$\$ORIGIN/$OPENSSL_RPATH'" ./configure ...

The OpenSSL bit is self-explanatory, the other one (../lib) is for finding libpython.so at runtime

I don't know if you're going all out and embedding ncurses, sqlite, tcl, etc. as well, but I can tell you that doing so is really exhausting... I used to have an RPM spec file that handled all of those - building them and installing them all together under a common directory with RUNPATH $ORIGIN set to find each but I left it behind at a former employer