[REQUEST] Windows surport

[Laser-Cho]

In WSL(Windows Subsystem Linux. it is just linux), this package works well.

But in Just normal windows, the simple code below is not run well but wait forever.
and terminal shows lot of errors about multiprocessing.

what i tried

import numpy as np
from parallelplot import parallel_plot

def scatter(data, fig, axes):
    axes.scatter(data[:, 0], data[:, 1])

X = np.random.uniform(low=-1, high=1, size=(5, 50, 50))

parallel_plot(plot_fn=scatter, data=X, grid_shape=(1, 5),  show_progress=False )

result in terminal is

Process SpawnPoolWorker-3:
Traceback (most recent call last):
  File "\Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "\Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "\Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "\Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>
Process SpawnPoolWorker-1:
Traceback (most recent call last):
  File "\Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "\Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "\Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "\Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-4:
Process SpawnPoolWorker-5:
Traceback (most recent call last):
  File "\Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "\Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "\Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
Traceback (most recent call last):
  File "\Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
  File "\Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>
  File "\Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "\Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "\Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "\Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>
  File "\Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "\Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "\Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>

Describe the solution you'd like
not sure, but...... probably multiprocessing is the key point.

[paulgavrikov]

hmm, yeah sounds like it could be related to multiprocessing, as it allocates workers differently on windows. Could you try setting mp_context with "fork", "forksever", or "spawn" and see if that fixes your issue?

[Laser-Cho]

sure.

fork

>>> parallel_plot(plot_fn=scatter, data=X, grid_shape=(1, 5),  show_progress=False , mp_context="fork")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "\WIN_VENV\lib\site-packages\parallelplot-0.0.4-py3.10.egg\parallelplot\plot.py", line 139, in parallel_plot
  File "\WIN_VENV\lib\site-packages\parallelplot-0.0.4-py3.10.egg\parallelplot\plot.py", line 60, in _make_subplots
  File "\Python310\lib\multiprocessing\context.py", line 239, in get_context
    return super().get_context(method)
  File "\Python310\lib\multiprocessing\context.py", line 193, in get_context
    raise ValueError('cannot find context for %r' % method) from None
ValueError: cannot find context for 'fork'

forksever

>>> parallel_plot(plot_fn=scatter, data=X, grid_shape=(1, 5),  show_progress=False, mp_context="forkserver")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "WIN_VENV\lib\site-packages\parallelplot-0.0.4-py3.10.egg\parallelplot\plot.py", line 139, in parallel_plot
  File "WIN_VENV\lib\site-packages\parallelplot-0.0.4-py3.10.egg\parallelplot\plot.py", line 60, in _make_subplots
  File "\Python310\lib\multiprocessing\context.py", line 239, in get_context
    return super().get_context(method)
  File "\Python310\lib\multiprocessing\context.py", line 193, in get_context
    raise ValueError('cannot find context for %r' % method) from None
ValueError: cannot find context for 'forkserver'
>>>

spawn

>>> parallel_plot(plot_fn=scatter, data=X, grid_shape=(1, 5),  show_progress=False, mp_context="spawn")
Process SpawnPoolWorker-1:
Traceback (most recent call last):
  File "Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-4:
Traceback (most recent call last):
  File "Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
Traceback (most recent call last):
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>
  File "Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>
Process SpawnPoolWorker-3:
Process SpawnPoolWorker-5:
Traceback (most recent call last):
Traceback (most recent call last):
  File "Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "Python310\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "Python310\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
  File "Python310\lib\multiprocessing\queues.py", line 367, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>
AttributeError: Can't get attribute 'scatter' on <module '__main__' (built-in)>

[Laser-Cho]

I solved this problem. in Windows, multiprocessing needs "main" . So beloe code wroks.

import numpy as np
from parallelplot import parallel_plot

def scatter(data, fig, axes):
    axes.scatter(data[:, 0], data[:, 1])

X = np.random.uniform(low=-1, high=1, size=(5, 100, 100))
if __name__=='__main__': 
    fig,axes = parallel_plot(plot_fn=scatter, data=X, grid_shape=(1, 5),  show_progress=False, mp_context="spawn" )
    fig.savefig("result.jpeg")

[Laser-Cho]

even I solved it, but.... I'm sorry to say like this, But it is just faster very slightly than normal ploting, even it use all cores. probably coping data makes new bottleneck.

[paulgavrikov]

There is of course some overhead associated with the allocation of workers and IO. The context itself also impacts performance. On Linux, I definitely saw a massive speedup.