Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wip: allow multiprocess dep instead of multiprocessing - 2nd attempt #100

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

lalo
Copy link
Contributor

@lalo lalo commented Mar 17, 2023

Not working, not sure if it's possible or if I'm missing something. Since the dependency happens during runtime, it is not able to fork. If it is decided during import time like the other PR - it doesn't need this extra un-picklable runtime object.

PS C:\Repos\pypeln> python3.9 .\examples\process_error.py       
pipeline:   0%|                                                                                                                                                                                                                                 | 0/300000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "C:\Repos\pypeln\examples\process_error.py", line 23, in <module>
    main()
  File "C:\Repos\pypeln\examples\process_error.py", line 20, in main
    pl.process.run(stage)
  File "C:\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypeln\process\api\run.py", line 42, in run
    for _ in stage:
  File "C:\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tqdm\std.py", line 1195, in __iter__
    for obj in iterable:
  File "C:\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypeln\process\stage.py", line 83, in to_iterable
    with supervisor:
  File "C:\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypeln\process\supervisor.py", line 39, in __enter__
    self.start()
  File "C:\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypeln\process\supervisor.py", line 53, in start
    worker.start()
  File "C:\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pypeln\process\worker.py", line 130, in start
    [self.process] = start_workers(self, use_threads=self.use_threads)
    t.start()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\process.py", line 121, in start
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\reduction.py", line 60, in dump
AttributeError: Can't pickle local object 'create_iterable_queue_class.<locals>.IterableQueue'
PS C:\Repos\pypeln> Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\multiprocessing\spawn.py", line 102, in spawn_main
    source_process = _winapi.OpenProcess(
OSError: [WinError 87] The parameter is incorrect

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant