Explore support for Python 3.15 limited API extensions

[ngoldbaum]

Python 3.15 will (hopefully) include a new ABI that allows building limited API extensions for both the GIL-enabled and free-threaded build. The new ABI will abstract over differences between the GIL-enabled and free-threaded ABI by making PyObject opaque.

See PEP 803 and PEP 809 for more details. There is currently experimental support in CPython for building PEP 803 stable ABI extensions via the private _Py_OPAQUE_PYOBJECT macro. I want to enable real-world end-to-end testing for the new stable ABI to allow the Python steering council to make informed and confident decisions.

It's likely the PyModuleDef usage in nd_defs.h will need updates to use the PyModExport API on Python 3.15 and newer. I also see a few structs that statically extend PyObject_HEAD in nanobind internals that may need to be converted to heap types.

It'll also require adding support for the Python 3.15 limited API. I see here there was some resistance to supporting targeting newer versions of the stable API.

See numpy/numpy#30704, python-cffi/cffi#226, and cython/cython#7399 for more information about this in other projects with widely used C APIs that are attempting to add support.

[ngoldbaum]

@wjakob maybe I could get a quote from you to add to PEP 803, if you're interested? python/peps#4831

[wjakob]

It's not in the stable ABI AFAIK.

[ngoldbaum]

I think I should have linked PyType_GetTypeDataSize

[ngoldbaum]

Also PyObject_GetTypeData.

Here's what PyO3 does: https://github.com/PyO3/pyo3/blob/c1f3d3442388cd93d2f01cad66139127390ca436/src/pycell/impl_.rs#L491

Sorry for the misdirection, it took me a little while to re-understand this again :)

[wjakob]

PyObject_GetTypeData does not seem to produce the right offset. I still need to add sizeof(size_t), which seems not so nice to depend on.

[ngoldbaum]

The conclusion here is that there is going to need to be a function in the stable ABI that does this without any external knowledge of the size of static types in CPython: https://discuss.python.org/t/pep-803-round-two-abi3t-stable-abi-for-free-threaded-builds/106181/18

[ngoldbaum]

@wjakob it looks like nanobind doesn't yet support the PyModExport or PyABIInfo APIs. Would it be helpful for someone else to add support? I did this pass already for PyO3 and I think I should be able to do a similar refactor here inside the NB_MODULE_IMPL2 macro.

[hpkfft]

Hi Nathan,
I think some initial investigation is in the abi3t branch.

[wjakob]

We need stable ABI support for PyVarObject element access. This is the key missing bit

[ngoldbaum]

Ah, I missed the code was already in a public spot. Thanks for the pointer.

[ngoldbaum]

@kumaraditya303 has a proposed workaround: #1324 for the issues with the 3.15 stable API that might allow a way to get nanobind working without waiting for 3.16.

[wjakob]

It would be a shame if 3.15 ships without a stable ABI for PyVarObject. I had stated in a few places that this would be a prerequisite for nanobind compatibility. The alternative workarounds are complex and have a nontrivial performance cost, which makes them incompatible with the goals of this project.