Expose functions called from the interpreter loop via PyAPI_FUNC

[martindemello]

Rationale: JIT code often needs access to functions called from the interpreter loop (in ceval.c or bytecodes.c), even if they are technically internal functions. gh-115802 is a good example - it added PyAPI_FUNC to a lot of functions in Include/internal.

Cinder currently relies on a fork of 3.12 with a lot of new functions exposed; some but not all of these have also been exposed in 3.13+. As a start, I would like to send in a PR for the functions we have already needed to expose for cinder (e.g. _PyNumber_InPlacePowerNoMod), but more generally it might be useful to look at all the Py* and _Py* functions used directly in ceval or bytecodes and consider exposing them even if no specific JIT has needed them yet.

Linked PRs

• gh-131776: Expose functions called from the interpreter loop via PyAP… #134242

[brandtbucher]

more generally it might be useful to look at all the Py* and _Py* functions used directly in ceval or bytecodes and consider exposing them even if no specific JIT has needed them yet.

Eh, I'm inclined to wait until people ask for them. Just exposing a bunch of internal, unstable functions with no users doesn't make much sense to me. At best, nobody will actually use them. At worst, people will start using them. :)

[ZeroIntensity]

If anything, it would be good to expose things in the unstable API rather than the private API.

[vstinner]

we have already needed to expose for cinder (e.g. _PyNumber_InPlacePowerNoMod)

If you have a list of functions that would be useful to Cinder, please give the list :-) We may expose these functions in the unstable API, as @ZeroIntensity suggests.

[martindemello]

here's the list of functions where (a) cinder exposed them as PyAPI_FUNC in 3.12, (b) cpython has not exposed them in main, and (c) the function still exists:

Include/internal/pycore_pyerrors.h
   _PyErr_GetRaisedException
   _PyErr_SetRaisedException

Include/internal/pycore_typeobject.h
   _PyStaticType_GetState

Include/internal/pycore_instruments.h
   _Py_call_instrumentation
   _Py_call_instrumentation_line
   _Py_call_instrumentation_instruction
   _Py_call_instrumentation_jump
   _Py_call_instrumentation_arg
   _Py_call_instrumentation_2args
   _Py_call_instrumentation_exc2

Include/internal/pycore_genobject.h
   _PyAsyncGenValueWrapperNew

Include/internal/pycore_ceval.h
   _Py_CheckRecursiveCallPy

Include/internal/pycore_dict.h
   _PyObjectDict_SetItem

Include/internal/pycore_abstract.h
   _PyNumber_PowerNoMod
   _PyNumber_InPlacePowerNoMod

Include/internal/pycore_call.h
   _PyStack_UnpackDict
   _PyStack_UnpackDict_FreeNoDecRef

Include/internal/pycore_code.h
   _PyCode_InitAddressRange
   _PyLineTable_NextAddressRange
   _Py_Specialize_LoadSuperAttr
   _Py_Specialize_LoadAttr
   _Py_Specialize_StoreAttr
   _Py_Specialize_LoadGlobal
   _Py_Specialize_StoreSubscr
   _Py_Specialize_Call
   _Py_Specialize_BinaryOp
   _Py_Specialize_CompareOp
   _Py_Specialize_UnpackSequence
   _Py_Specialize_ForIter
   _Py_Specialize_Send
   _Py_Instrument

[vstinner]

Hum, most of these functions are low-level and the internal C API is the right place for them. I'm not sure about adding PyAPI_FUNC() to them.

For now, I would suggest to continue maintaining downstream patches on the Cinder fork to add these PyAPI_FUNC().

I'm not comfortable to add these functions to the public C API.

_PyErr_GetRaisedException
_PyErr_SetRaisedException

Compared to PyErr_GetRaisedException() and PyErr_SetRaisedException(), passing tstate is an optimization. So far, we didn't expose "Tstate" variants to the public C API. There are reserved to the internal C API such as _PyErr_Occurred() (implemented as a static inline function).

_PyStaticType_GetState

I would prefer to not expose managed_static_type_state structure which is low-level.

_PyNumber_PowerNoMod
_PyNumber_InPlacePowerNoMod

You may reimplement these functions in Cinder, there are trivial:

PyObject *
_PyNumber_PowerNoMod(PyObject *lhs, PyObject *rhs)
{   
    return PyNumber_Power(lhs, rhs, Py_None);
}   

PyObject *
_PyNumber_InPlacePowerNoMod(PyObject *lhs, PyObject *rhs)
{   
    return PyNumber_InPlacePower(lhs, rhs, Py_None);
}   

[brandtbucher]

Just to be clear, these would remain in the internal C API. PyAPI_FUNC would just be added so it's at least possible to link against them for projects that want to include the internal headers.

[martindemello]

yes, all we need is to be able to link against it, not any sort of stability guarantees. I like the idea of an alias of PyAPI_FUNC that makes this explicit.

[DinoV]

Instead of exposing _PyStaticType_GetState I think we could expose lookup_tp_subclasses and init_tp_subclasses (obviously under new names... we also use it to get the dict, but _PyType_GetDict is exported).

Not having _PyErr_SetRaisedException and _PyErr_GetRaisedException is probably fine, although we do use it in the same place that CPython is using it (which is in the interpreter loop).

+1 on just re-implementing _PyNumber_PowerNoMod and _PyNumber_InPlacePowerNoMod.