Segfaults with frequent python calls from thread 1 on multithreaded Julia

[oschulz]

In a complex application with high-frequency python calls I always run into

signal (11): Segmentation fault
in expression starting at REPL[41]:2
_PyTrash_begin at /opt/conda/conda-bld/python-split_1649141344976/work/Objects/object.c:2125

segfaults when Julia has multiple threads (e.g. JULIA_NUM_THREADS=4). I do make sure to only use PythonCall from thread 1 (and I use asserts to verify that) but somehow that doesn't seem to be enough. When running with JULIA_NUM_THREADS=1, all is fine. Is PythonCall incompatible with multi-threaded Julia in general?

[cjdoris]

I have never really tried it out, but I'm not too surprised.

AFAIK it should work fine provided all the calls into the CPython API occur from the same thread (or are guarded with PyGILState_Ensure) which includes not just user code, but also the __init__ function (which initialises the Python interpreter) and the finalizer for Py (which frees Python objects). It sounds like the init/finalizer functions will need to be modified to allow them to run on other threads even if user code is all on thread 1.

Do you have a MWE I can try?

[oschulz]

Do you have a MWE I can try?

At the moment, it's definitely not minimal, but I'll see what I can cook up.

(or are guarded with PyGILState_Ensure)

Oh, how do I do that?

the finalizer for Py (which frees Python objects).

I had a feeling it might be related to that - can I control where the finalizer runs?

[cjdoris]

See #202 (comment)

If you try out that branch and put PythonCall.C.gc_disable() and PythonCall.C.gc_enable() around the threaded part of your code, maybe it will work?

[oschulz]

around the threaded part of your code, maybe it will work?

The problem is, Python is calling an MCMC sampler written in Julia, which then calls back to evaluate the actual target function in Python. So I'd have to disable Python GC for a very long time, memory will overflow. The Julia code itself does container multithreaded operations, though I've made sure that the callbacks to Python are always originating from thread 1. Other parts of the Julia code (not calling back into Python) will use threads, but from Python, only a single algorithm is called, I I can't "inject" gc-disable/enables into the Julia code in lots of places.

[cjdoris]

I've just released a new version including these GC functions, they are now called PythonCall.GC.[dis/en]able.

In your case, you should be able to do the whole optimization with GC disabled, but temporarily re-enable it within the callback function.

[oschulz]

Thanks a lot @cjdoris , will try it out!

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

[github-actions]

This issue has been closed because it has been stale for 7 days. If it is still
relevant, please re-open it.

Why not just check in the enable function if the current thread is the main thread and skip GC if that's the case?