`ProcessPoolExecutor` raises exception or hangs the process when `shutdown(wait=False)` and worker throws exception

[ogbiggles]

Bug report

Bug description:

Below code demonstrates two problems that occur when
(a) worker process throws an exception, and
(b) shutdown(wait=False) is used

1. When max_workers < number of submitted tasks, raises exception TypeError: object of type 'NoneType' has no len()
2. When max_workers > number of submitted tasks, raises exception TypeError: object of type 'NoneType' has no len() AND then program hangs

There is no problem when shutdown(wait=True) is used or if the worker processes exit cleanly.

import concurrent.futures
import multiprocessing as mp
import time


def task(n: int) -> int:
    if n == 2:
        raise Exception("Not gonna do it")
    else:
        time.sleep(n)
        return n


def main() -> None:
    # max_workers=2 results in raised exception and program exits
    # max_workers=4 results in raised exception and program hangs
    #
    executor = concurrent.futures.ProcessPoolExecutor(
        max_workers=2, mp_context=mp.get_context("forkserver"), max_tasks_per_child=1
    )
    f1 = executor.submit(task, 1)
    f2 = executor.submit(task, 2)
    f3 = executor.submit(task, 3)
    result = 0
    try:
        result += f1.result()
        result += f2.result()
        result += f3.result()
        print(f"Result = {result}")
    except Exception as e:
        print(f"Exception while getting result : {e}")

    executor.shutdown(wait=False)


if __name__ == "__main__":
    main()

CPython versions tested on:

3.12

Operating systems tested on:

Linux, macOS

Linked PRs

• [3.12] gh-132969: Fix exception/hang shutdown(wait=False) and a task exited abnormally #133220
• gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally #133222

[YvesDup]

After deeping in code, it seems to me that the bug is the run method of _ExecutorManagerThread.
When shutdown with wait set to False is called, this method always tries to ajust the process count.

cpython/Lib/concurrent/futures/process.py

Lines 361 to 363 in 8b26b23

	if process_exited:
	with self.shutdown_lock:
	executor._adjust_process_count()

These lines should be replace as below:

         if process_exited:
              if not self.is_shutting_down():
                  with self.shutdown_lock:
                      executor._adjust_process_count()

This modification fixes the 2 cases

[ogbiggles]

Shutdown wait=False resets the ProcessPoolExecutor's state, hence returning from _adjust_process_counts() when the state has been reset fixes both issues. I submitted a PR #133220 with that fix and a couple of unittests to validate the fix.

p.s. my PR is on the 3.12 branch, and this is my first time contributing to cpython, so am not sure what steps (if any) I need to follow for porting the fix to 3.13 and main.

[ogbiggles]

p.s. my PR is on the 3.12 branch, and this is my first time contributing to cpython, so am not sure what steps (if any) I need to follow for porting the fix to 3.13 and main.

Resubmitted a new pull request on main since the 3.12 branch is closed for bug fixes.