Usage with distributed

[marius311]

I'm trying to pmap over a simple Python function, not trying to move any Python objects between processes or any intercommunication otherwise, but getting a segfault. Is this supposed to work? Or a way to make it work?

$ julia -p 2

julia> using PythonCall

julia> @py 1+2
Python: 3

julia> pmap(1:2) do i
           @py 1+2
           nothing
       end

# segfault

Julia 1.9.3, PythonCall 0.9.14, python 3.11.5.

[marius311]

I think the issue is that @py is expanded on the main process, which would be fine except the macro pastes in an actual pointer (py NULL) into the expression which means its trying to get shipped to the workers:

julia> @macroexpand @py 1+2
quote
    #= REPL[11]:1 =#
    if (Base.RefValue{Bool}(true))[]
        (PythonCall.pycopy!)(<py NULL>, (PythonCall.pynew)((Py)(2)))
        (PythonCall.pycopy!)(<py NULL>, (PythonCall.pynew)((Py)(1)))
        (Base.RefValue{Bool}(true))[] = false
    end
    var"##x#320" = <py NULL>
    var"##y#321" = <py NULL>
    var"##ans#319" = (PythonCall.pyadd)(var"##x#320", var"##y#321")
    var"##ans#319"
end

Doing it this way expands the marco on the worker, which seems to be fine

julia> pmap(1:2) do i
           @eval @py 1+2
           nothing
       end
2-element Vector{Nothing}:
 nothing
 nothing

Don't know if its worth doing anything. The macro could expand to a function call which returns PyNull rather than pasting the object in there?

[cjdoris]

I've never tried PythonCall with Distributed. Does it work if you avoid @py but do other Python things such as pyadd(1, 2)?