ProcessPoolExecutor should not share one BrokenProcessPool exception among all futures

[daniel-shields]

Summary

For Python 3.9.x through what's on main now,

When a worker process terminates abruptly, concurrent.futures.ProcessPoolExecutor creates a single BrokenProcessPool exception to share among all futures (code). This is unsafe because exceptions are mutable.

As a consequence, calling Future.result() on all failed futures will result in malformed tracebacks. Each traceback will be the concatenation of all prior tracebacks with the current one.

This issue can be resolved by creating a separate exception instance for each failed future.

Note that in the example below, the size of the malformed traceback is limited. In real-world applications, it can easily be thousands of lines.

Example Code

from concurrent.futures import ProcessPoolExecutor
import time
import traceback


def main():
    with ProcessPoolExecutor() as e:
        futures = [e.submit(time.sleep, 10) for _ in range(3)]
        # Get one of the processes, and terminate (kill) it.
        p = next(iter(e._processes.values()))
        p.terminate()
        for i, fut in enumerate(futures):
            try:
                fut.result()
            except Exception as e:
                print(f"\nException {i}:")
                traceback.print_exception(e)

if __name__ == "__main__":
    main()

Current (Incorrect) Output

Notice that each consecutive traceback appends to the previous one.

Exception 0:
Traceback (most recent call last):
  File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
    fut.result()
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Exception 1:
Traceback (most recent call last):
  File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
    fut.result()
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
    fut.result()
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Exception 2:
Traceback (most recent call last):
  File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
    fut.result()
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
    fut.result()
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
    fut.result()
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

New (Correct) Output

Exception 0:
Traceback (most recent call last):
  File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
    fut.result()
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Exception 1:
Traceback (most recent call last):
  File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
    fut.result()
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Exception 2:
Traceback (most recent call last):
  File "C:\Users\Daniel\open_source\junk\example.py", line 14, in main
    fut.result()
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Daniel\open_source\cpython\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Linked PRs

• gh-101267: [bug] ProcessPoolExecutor should not share one BrokenProcessPool exception among all failed futures #101268