test_remote_pdb hangs randomly

[vstinner]

Example on "Hypothesis tests on Ubuntu": https://github.com/python/cpython/actions/runs/14660752238/job/41144501229?pr=132906

(...)
0:46:29 load avg: 0.00 running (1): test_remote_pdb (43 min 3 sec)
0:46:59 load avg: 0.00 running (1): test_remote_pdb (43 min 33 sec)
0:47:29 load avg: 0.00 running (1): test_remote_pdb (44 min 3 sec)
0:47:59 load avg: 0.00 running (1): test_remote_pdb (44 min 33 sec)
0:48:29 load avg: 0.00 running (1): test_remote_pdb (45 min 3 sec)
0:48:59 load avg: 0.00 running (1): test_remote_pdb (45 min 33 sec)

Sadly, the "Hypothesis tests on Ubuntu" job has no timeout :-(

Linked PRs

• gh-132912: Set 15 min timeout on GHA Hypothesis CI #132914
• gh-132912: Kill the process on error in test_remote_pdb #132920
• gh-132912: Skip flaky test in test_remote_pdb #132924
• gh-132912: Account for race in test_keyboard_interrupt in test_remote_pdb #132929
• [3.13] gh-132912: Set 15 min timeout on GHA Hypothesis CI (GH-132914) #132937
• gh-132912: Use SHORT_TIMEOUT in test_remote_pdb #132939
• gh-132912: Use readline to synchronize between procs in remote pdb test #132949

[picnixz]

The test has been added in 797b29b (cc @pablogsal @godlygeek)

[vstinner]

I reproduced an issue locally on Linux:

$ ./python -m test test_remote_pdb -v --forever -j10 --timeout=60
...
0:02:56 load avg: 7.49 [242/1] test_remote_pdb worker non-zero exit code (Exit code 1)
test_breakpoints (test.test_remote_pdb.PdbConnectTestCase.test_breakpoints)
Test setting and hitting breakpoints. ... ok
test_connect_and_basic_commands (test.test_remote_pdb.PdbConnectTestCase.test_connect_and_basic_commands)
Test connecting to a remote debugger and sending basic commands. ... ok
test_handle_eof (test.test_remote_pdb.PdbConnectTestCase.test_handle_eof)
Test that EOF signal properly exits the debugger. ... ok
test_help_system (test.test_remote_pdb.PdbConnectTestCase.test_help_system)
Test that the help system properly sends help text to the client. ... ok
test_keyboard_interrupt (test.test_remote_pdb.PdbConnectTestCase.test_keyboard_interrupt)
Test that sending keyboard interrupt breaks into pdb. ... Timeout (0:01:00)!
Thread 0x00007fc086c20780 [python] (most recent call first):
  File "/home/vstinner/python/main/Lib/subprocess.py", line 2019 in _try_wait
  File "/home/vstinner/python/main/Lib/subprocess.py", line 2061 in _wait
  File "/home/vstinner/python/main/Lib/subprocess.py", line 1275 in wait
  File "/home/vstinner/python/main/Lib/subprocess.py", line 1129 in __exit__
  File "/home/vstinner/python/main/Lib/test/test_remote_pdb.py", line 492 in test_keyboard_interrupt
  (...)
  File "/home/vstinner/python/main/Lib/runpy.py", line 198 in _run_module_as_main
...

[vstinner]

The problem comes from the with process: statement. If a test fails, it waits until the process completes. But the child process does not complete, and so Python hangs.

[vstinner]

With a local fix for with process:, I get an error message instead:

======================================================================
FAIL: test_keyboard_interrupt (test.test_remote_pdb.PdbConnectTestCase.test_keyboard_interrupt)
Test that sending keyboard interrupt breaks into pdb.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_remote_pdb.py", line 509, in test_keyboard_interrupt
    self.assertIn("bar()", interrupt_msg)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'bar()' not found in '> /home/vstinner/python/main/Lib/socket.py(855)create_connection()\n-> sock.connect(sa)\n'

[pablogsal]

I will take a look soon

[vstinner]

The problem comes from the with process: statement. If a test fails, it waits until the process completes. But the child process does not complete, and so Python hangs.

I wrote #132920 to prevent test_remote_pdb to hang: it kills the child process on error.