gh-87135: Hang non-main threads that attempt to acquire the GIL during finalization

Doc/c-api/init.rst

@@ -409,7 +409,11 @@ Initializing and finalizing the interpreter
    freed.  Some memory allocated by extension modules may not be freed.  Some
    extensions may not work properly if their initialization routine is called more
    than once; this can happen if an application calls :c:func:`Py_Initialize` and
-   :c:func:`Py_FinalizeEx` more than once.
+   :c:func:`Py_FinalizeEx` more than once.  :c:func:`Py_FinalizeEx` must not be
+   called recursively from within itself.  Therefore, it must not be called by any
+   code that may be run as part of the interpreter shutdown process, such as
+   :py:mod:`atexit` handlers, object finalizers, or any code that may be run while
+   flushing the stdout and stderr files.
 
    .. audit-event:: cpython._PySys_ClearAuditHooks "" c.Py_FinalizeEx
 
@@ -1000,6 +1004,78 @@ thread, where the CPython global runtime was originally initialized.
 The only exception is if :c:func:`exec` will be called immediately
 after.
 
+.. _cautions-regarding-runtime-finalization:
+
+Cautions regarding runtime finalization
+---------------------------------------
+
+In the late stage of :term:`interpreter shutdown`, after attempting to wait for
+non-daemon threads to exit (though this can be interrupted by
+:class:`KeyboardInterrupt`) and running the :mod:`atexit` functions, the runtime
+is marked as *finalizing*: :c:func:`_Py_IsFinalizing` and
+:func:`sys.is_finalizing` return true.  At this point, only the *finalization
+thread* that initiated finalization (typically the main thread) is allowed to
+acquire the :term:`GIL`.
+
+If any thread, other than the finalization thread, attempts to acquire the GIL
+during finalization, either explicitly via :c:func:`PyGILState_Ensure`,
+:c:macro:`Py_END_ALLOW_THREADS`, :c:func:`PyEval_AcquireThread`, or
+:c:func:`PyEval_AcquireLock`, or implicitly when the interpreter attempts to
+reacquire it after having yielded it, the thread enters a permanently blocked
+state where it remains until the program exits.  In most cases this is harmless,
+but this can result in deadlock if a later stage of finalization attempts to
+acquire a lock owned by the blocked thread, or otherwise waits on the blocked
+thread.
+
+To avoid non-Python threads becoming blocked, or Python-created threads becoming
+blocked while executing C extension code, you can use
+:c:func:`PyThread_TryAcquireFinalizeBlock` and
+:c:func:`PyThread_ReleaseFinalizeBlock`.
+
+For example, to deliver an asynchronous notification to Python from a C
+extension, you might be inclined to write the following code that is *not* safe
+to execute during finalization:
+
+.. code-block:: c
+
+   // some non-Python created thread that wants to send Python an async notification
+   PyGILState_STATE state = PyGILState_Ensure(); // may hang thread
+   // call `call_soon_threadsafe` on some event loop object
+   PyGILState_Release(state);
+
+To avoid the possibility of the thread hanging during finalization, and also
+support older Python versions:
+
+.. code-block:: c
+
+   // some non-Python created thread that wants to send Python an async notification
+   PyGILState_STATE state;
+   #if PY_VERSION_HEX >= 0x030c0000 // API added in Python 3.12
+   int acquired = PyThread_TryAcquireFinalizeBlock();
+   if (!acquired) {
+     // skip sending notification since python is exiting
+     return;
+   }
+   #endif // PY_VERSION_HEX
+   state = PyGILState_Ensure(); // safe now
+   // call `call_soon_threadsafe` on some event loop object
+   PyGILState_Release(state);
+   #if PY_VERSION_HEX >= 0x030c0000 // API added in Python 3.12
+   PyThread_ReleaseFinalizeBlock();
+   #endif // PY_VERSION_HEX
+
+Or with the convenience interface (requires Python >=3.12):
+
+.. code-block:: c
+
+   // some non-Python created thread that wants to send Python an async notification
+   PyGILState_TRY_STATE state = PyGILState_TryAcquireFinalizeBlockAndGIL();
+   if (!state) {
+     // skip sending notification since python is exiting
+     return;
+   }
+   // call `call_soon_threadsafe` on some event loop object
+   PyGILState_ReleaseGILAndFinalizeBlock(state);
 
 High-level API
 --------------
@@ -1082,11 +1158,14 @@ code, or when embedding the Python interpreter:
    ensues.
 
    .. note::
-      Calling this function from a thread when the runtime is finalizing
-      will terminate the thread, even if the thread was not created by Python.
-      You can use :c:func:`_Py_IsFinalizing` or :func:`sys.is_finalizing` to
-      check if the interpreter is in process of being finalized before calling
-      this function to avoid unwanted termination.
+      Calling this function from a thread when the runtime is finalizing will
+      hang the thread until the program exits, even if the thread was not
+      created by Python.  Refer to
+      :ref:`cautions-regarding-runtime-finalization` for more details.
+
+   .. versionchanged:: 3.12
+      Hangs the current thread, rather than terminating it, if called while the
+      interpreter is finalizing.
 
 .. c:function:: PyThreadState* PyThreadState_Get()
 
@@ -1128,11 +1207,14 @@ with sub-interpreters:
    to call arbitrary Python code.  Failure is a fatal error.
 
    .. note::
-      Calling this function from a thread when the runtime is finalizing
-      will terminate the thread, even if the thread was not created by Python.
-      You can use :c:func:`_Py_IsFinalizing` or :func:`sys.is_finalizing` to
-      check if the interpreter is in process of being finalized before calling
-      this function to avoid unwanted termination.
+      Calling this function from a thread when the runtime is finalizing will
+      hang the thread until the program exits, even if the thread was not
+      created by Python.  Refer to
+      :ref:`cautions-regarding-runtime-finalization` for more details.
+
+   .. versionchanged:: 3.12
+      Hangs the current thread, rather than terminating it, if called while the
+      interpreter is finalizing.
 
 .. c:function:: void PyGILState_Release(PyGILState_STATE)
 
@@ -1144,6 +1226,36 @@ with sub-interpreters:
    Every call to :c:func:`PyGILState_Ensure` must be matched by a call to
    :c:func:`PyGILState_Release` on the same thread.
 
+.. c:function:: PyGILState_TRY_STATE PyGILState_AcquireFinalizeBlockAndGIL()
+
+   Attempts to acquire a :ref:`finalize
+   block<cautions-regarding-runtime-finalization>`, and if successful, acquires
+   the :term:`GIL`.
+
+   This is a simple convenience interface that saves having to call
+   :c:func:`PyThread_TryAcquireFinalizeBlock` and :c:func:`PyGILState_Ensure`
+   separately.
+
+   Returns ``PyGILState_TRY_LOCK_FAILED`` (equal to 0) if the interpreter is
+   already waiting to finalize.  In this case, the :term:`GIL` is not acquired
+   and Python C APIs that require the :term:`GIL` must not be called.
+
+   Otherwise, acquires a finalize block and then acquires the :term:`GIL`.
+
+   Each call that is successful (i.e. returns a non-zero
+   ``PyGILState_TRY_STATE`` value) must be paired with a subsequent call to
+   :c:func:`PyGILState_ReleaseGILAndFinalizeBlock` with the same value returned
+   by this function.  Calling :c:func:`PyGILState_ReleaseGILAndFinalizeBlock` with the
+   error value ``PyGILState_TRY_LOCK_FAILED`` is safe and does nothing.
+
+   .. versionadded:: 3.12
+
+.. c:function:: void PyGILState_ReleaseGILAndFinalizeBlock(PyGILState_TRY_STATE)
+
+   Releases any locks acquired by the corresponding call to
+   :c:func:`PyGILState_AcquireFinalizeBlockAndGIL`.
+
+   .. versionadded:: 3.12
 
 .. c:function:: PyThreadState* PyGILState_GetThisThreadState()
 
@@ -1410,17 +1522,20 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
    If this thread already has the lock, deadlock ensues.
 
    .. note::
-      Calling this function from a thread when the runtime is finalizing
-      will terminate the thread, even if the thread was not created by Python.
-      You can use :c:func:`_Py_IsFinalizing` or :func:`sys.is_finalizing` to
-      check if the interpreter is in process of being finalized before calling
-      this function to avoid unwanted termination.
+      Calling this function from a thread when the runtime is finalizing will
+      hang the thread until the program exits, even if the thread was not
+      created by Python.  Refer to
+      :ref:`cautions-regarding-runtime-finalization` for more details.
 
    .. versionchanged:: 3.8
       Updated to be consistent with :c:func:`PyEval_RestoreThread`,
       :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`,
       and terminate the current thread if called while the interpreter is finalizing.
 
+   .. versionchanged:: 3.12
+      Hangs the current thread, rather than terminating it, if called while the
+      interpreter is finalizing.
+
    :c:func:`PyEval_RestoreThread` is a higher-level function which is always
    available (even when threads have not been initialized).
 
@@ -1448,17 +1563,19 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
       instead.
 
    .. note::
-      Calling this function from a thread when the runtime is finalizing
-      will terminate the thread, even if the thread was not created by Python.
-      You can use :c:func:`_Py_IsFinalizing` or :func:`sys.is_finalizing` to
-      check if the interpreter is in process of being finalized before calling
-      this function to avoid unwanted termination.
+      Calling this function from a thread when the runtime is finalizing will
+      hang the thread until the program exits, even if the thread was not
+      created by Python.  Refer to
+      :ref:`cautions-regarding-runtime-finalization` for more details.
 
    .. versionchanged:: 3.8
       Updated to be consistent with :c:func:`PyEval_RestoreThread`,
       :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`,
       and terminate the current thread if called while the interpreter is finalizing.
 
+   .. versionchanged:: 3.12
+      Hangs the current thread, rather than terminating it, if called while the
+      interpreter is finalizing.
 
 .. c:function:: void PyEval_ReleaseLock()
 
@@ -1469,6 +1586,32 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
       :c:func:`PyEval_SaveThread` or :c:func:`PyEval_ReleaseThread`
       instead.
 
+.. c:function:: int PyThread_AcquireFinalizeBlock()
+
+   Attempts to acquire a block on Python finalization.
+
+   While the *finalize block* is held, the Python interpreter will block before
+   it begins finalization.  Holding a finalize block ensures that the
+   :term:`GIL` can be safely acquired without the risk of hanging the thread.
+   Refer to :ref:`cautions-regarding-runtime-finalization` for more details.
+
+   If successful, returns 1.  If the interpreter is already finalizing, or about
+   to begin finalization and waiting for all previously-acquired finalize blocks
+   to be released, returns 0 without acquiring a finalize block.
+
+   Every successful call must be paired with a call to
+   :c:func:`PyThread_ReleaseFinalizeBlock`.
+
+   This function may be safely called with or without holding the :term:`GIL`.
+
+   .. versionadded:: 3.12
+
+.. c:function:: void PyThread_ReleaseFinalizeBlock()
+
+   Releases a finalize block acquired by a prior successful call to
+   :c:func:`PyThread_AcquireFinalizeBlock` (return value of 1).
+
+   .. versionadded:: 3.12
 
 .. _sub-interpreter-support:
 
@@ -2007,4 +2150,3 @@ be used in new code.
 .. c:function:: void* PyThread_get_key_value(int key)
 .. c:function:: void PyThread_delete_key_value(int key)
 .. c:function:: void PyThread_ReInitTLS()
-

Include/internal/pycore_runtime.h

@@ -84,6 +84,19 @@ typedef struct pyruntimestate {
        to access it, don't access it directly. */
     _Py_atomic_address _finalizing;
 
+    /* Tracks the finalize blocks.
+
+       Bit 0 is set to 1 by `Py_FinalizeEx` to indicate it is waiting to set `_finalizing`.
+
+       The remaining bits are a count of the number of finalize blocks that are
+       currently held.  Once bit 0 is set to 1, the number of finalize blocks is
+       not allowed to increase.
+
+       Protected by the main interpreter's GIL `main_interp->ceval.gil->mutex`;
+       `main_interp->ceval.gil->cond` must be broadcast when it becomes 1.
+    */
+    unsigned long finalize_blocks;
+
     struct _pymem_allocators allocators;
     struct _obmalloc_global_state obmalloc;
     struct pyhash_runtime_state pyhash_state;

Include/pystate.h

@@ -119,6 +119,53 @@ PyAPI_FUNC(void) PyGILState_Release(PyGILState_STATE);
 */
 PyAPI_FUNC(PyThreadState *) PyGILState_GetThisThreadState(void);
 
+/* Attempts to acquire a block on interpreter finalization.
+
+   Returns 1 on success, or 0 if the interpreter is already waiting to finalize.
+
+   While the lock is held, the interpreter will not enter the finalization
+   state.
+
+   Each call that returns 1 must be paired with a subsequent call to
+   `PyThread_ReleaseFinalizeBlock`.
+
+   It is not necessary to hold the GIL.  While holding a block on interpreter
+   finalization, a non-main thread can safely acquire the GIL without risking
+   becoming permanently blocked.
+ */
+PyAPI_FUNC(int) PyThread_TryAcquireFinalizeBlock(void);
+
+/* Releases the block acquired by a successful call to
+   `PyThread_TryAcquireFinalizeBlock`. */
+PyAPI_FUNC(void) PyThread_ReleaseFinalizeBlock(void);
+
+typedef enum {
+  PyGILState_TRY_LOCK_FAILED,
+  PyGILState_TRY_LOCK_LOCKED,
+  PyGILState_TRY_LOCK_UNLOCKED
+} PyGILState_TRY_STATE;
+
+/* Attempts to acquire a finalize block, and if successful, acquires the GIL.
+
+   This is a simple convenience interface that saves having to call
+   `PyThread_TryAcquireFinalizeBlock()` and `PyGILState_Ensure()` separately.
+
+   Returns `PyGILState_TRY_LOCK_FAILED` (equal to 0) if the interpreter is
+   already waiting to finalize.  In this case, the GIL is not acquired and
+   Python C APIs that require the GIL must not be called.
+
+   Otherwise, acquires a finalize block and then acquires the GIL.
+
+   Each call that is successful (i.e. returns a non-zero `PyGILState_TRY_STATE`
+   value) must be paired with a subsequent call to
+   `PyGILState_ReleaseGILAndFinalizeBlock` with the same value returned by this
+   function.  Calling `PyGILState_ReleaseGILAndFinalizeBlock` with the error
+   value `PyGILState_TRY_LOCK_FAILED` is safe and does nothing. */
+PyAPI_FUNC(PyGILState_TRY_STATE) PyGILState_TryAcquireFinalizeBlockAndGIL(void);
+
+/* Releases any locks acquired by the corresponding call to
+   `PyGILState_TryAcquireFinalizeBlockAndGIL`. */
+PyAPI_FUNC(void) PyGILState_ReleaseGILAndFinalizeBlock(PyGILState_TRY_STATE);
 
 #ifndef Py_LIMITED_API
 #  define Py_CPYTHON_PYSTATE_H

Include/pythread.h

@@ -17,7 +17,37 @@ typedef enum PyLockStatus {
 
 PyAPI_FUNC(void) PyThread_init_thread(void);
 PyAPI_FUNC(unsigned long) PyThread_start_new_thread(void (*)(void *), void *);
-PyAPI_FUNC(void) _Py_NO_RETURN PyThread_exit_thread(void);
+/* Terminates the current thread.
+ *
+ * WARNING: This function is only safe to call if all functions in the full call
+ * stack are written to safely allow it.  Additionally, the behavior is
+ * platform-dependent.  This function should be avoided, and is no longer called
+ * by Python itself.  It is retained only for compatibility with existing C
+ * extension code.
+ *
+ * With pthreads, calls `pthread_exit` which attempts to unwind the stack and
+ * call C++ destructors.  If a `noexcept` function is reached, the program is
+ * terminated.
+ *
+ * On Windows, calls `_endthreadex` which kills the thread without calling C++
+ * destructors.
+ *
+ * In either case there is a risk of invalid references remaining to data on the
+ * thread stack.
+ */
+Py_DEPRECATED(3.12) PyAPI_FUNC(void) _Py_NO_RETURN PyThread_exit_thread(void);
+
+#ifndef Py_LIMITED_API
+/* Hangs the thread indefinitely without exiting it.
+ *
+ * bpo-42969: There is no safe way to exit a thread other than returning
+ * normally from its start function.  This is used during finalization in lieu
+ * of actually exiting the thread.  Since the program is expected to terminate
+ * soon anyway, it does not matter if the thread stack stays around until then.
+ */
+PyAPI_FUNC(void) _Py_NO_RETURN _PyThread_hang_thread(void);
+#endif  /* !Py_LIMITED_API */
+
 PyAPI_FUNC(unsigned long) PyThread_get_thread_ident(void);
 
 #if (defined(__APPLE__) || defined(__linux__) || defined(_WIN32) \