Scanning `ArrowArrayStream` twice raises `InternalException: INTERNAL Error: ArrowArrayStream was released by another thread/library`

[henryharbeck]

What happens?

When scanning an ArrowArrayStream twice, an internal error is raised.

Per https://duckdb.org/docs/stable/dev/internal_errors this should be raised as an issue.

Potentially related to #70

To Reproduce

import duckdb        # 1.34.0
import polars as pl  # 1.4.0

df = pl.DataFrame({"a": 1})
cap = df.__arrow_c_stream__()
rel = duckdb.query("SELECT * from cap")

# Printing this relation causes an error in .pl() below
print(rel)

# _duckdb.InternalException: INTERNAL Error: ArrowArrayStream was released by another thread/library
print(rel.pl())

Output

┌───────┐
│   a   │
│ int64 │
├───────┤
│     1 │
└───────┘

Traceback (most recent call last):
  File "/home/henry/development/adbc-polars-duck-bq-pg/duckdb_polars_no_pyarrow/duckdb_polars_error.py", line 23, in <module>
    print(rel.pl())
          ~~~~~~^^
_duckdb.InternalException: INTERNAL Error: ArrowArrayStream was released by another thread/library

Stack Trace:

/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb9Exception6ToJSONENS_13ExceptionTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x56) [0x79b7e0978f86]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb9ExceptionC2ENS_13ExceptionTypeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x14) [0x79b7e0979034]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb17InternalExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x11) [0x79b7e097c131]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(+0x7c49f1) [0x79b7dffc49f1]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb34PythonTableArrowArrayStreamFactory9GetSchemaEmRNS_18ArrowSchemaWrapperE+0x25) [0x79b7e04754f5]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb18ArrowTableFunction13ArrowScanBindERNS_13ClientContextERNS_22TableFunctionBindInputERNS_6vectorINS_11LogicalTypeELb1EEERNS5_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEELb1EEE+0x2a0) [0x79b7e072c740]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb18ArrowTableFunction17ArrowScanBindDumbERNS_13ClientContextERNS_22TableFunctionBindInputERNS_6vectorINS_11LogicalTypeELb1EEERNS5_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEELb1EEE+0x12) [0x79b7e072c8c2]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder25BindTableFunctionInternalERNS_13TableFunctionERKNS_16TableFunctionRefENS_6vectorINS_5ValueELb1EEESt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_NS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIKSF_S7_EEENS6_INS_11LogicalTypeELb1EEENS6_ISF_Lb1EEE+0x567) [0x79b7e0fd55f7]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_16TableFunctionRefE+0xd68) [0x79b7e0fd8668]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_8TableRefE+0x128) [0x79b7e0526a38]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder8BindNodeERNS_10SelectNodeE+0x3b) [0x79b7e0f7c1db]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder8BindNodeERNS_9QueryNodeE+0x48) [0x79b7e0527988]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder7BindCTEERNS_7CTENodeE+0x1ae) [0x79b7e0f7173e]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder8BindNodeERNS_7CTENodeE+0x12) [0x79b7e0f72c02]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder8BindNodeERNS_9QueryNodeE+0x96) [0x79b7e05279d6]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_9QueryNodeE+0x2f) [0x79b7e0527a4f]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_15SelectStatementE+0x3a) [0x79b7e0f85b1a]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_12SQLStatementE+0x22d) [0x79b7e052792d]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb8Relation4BindERNS_6BinderE+0xea) [0x79b7e0cb79aa]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13QueryRelation4BindERNS_6BinderE+0x5e) [0x79b7e0c8be8e]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_17RelationStatementE+0x24) [0x79b7e0f85ad4]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb6Binder4BindERNS_12SQLStatementE+0xc5) [0x79b7e05277c5]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb7Planner10CreatePlanERNS_12SQLStatementE+0xa5) [0x79b7e05345a5]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext31CreatePreparedStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEENS_12optional_ptrISt13unordered_mapIS8_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIS9_SI_EEELb1EEE+0x28b) [0x79b7e0cdca6b]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext23CreatePreparedStatementERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEENS_12optional_ptrISt13unordered_mapIS8_NS_18BoundParameterDataENS_33CaseInsensitiveStringHashFunctionENS_29CaseInsensitiveStringEqualityESaISt4pairIS9_SI_EEELb1EEENS_21PreparedStatementModeE+0x347) [0x79b7e0cdd807]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext24PendingStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERKNS_22PendingQueryParametersE+0xa4) [0x79b7e0ceaf84]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext35PendingStatementOrPreparedStatementERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE+0xa0) [0x79b7e0cf2fa0]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext43PendingStatementOrPreparedStatementInternalERNS_17ClientContextLockERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEENS_10unique_ptrINS_12SQLStatementESt14default_deleteISC_ELb1EEERNS_10shared_ptrINS_21PreparedStatementDataELb1EEERKNS_22PendingQueryParametersE+0x85) [0x79b7e0cf31c5]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext20PendingQueryInternalERNS_17ClientContextLockENS_10unique_ptrINS_12SQLStatementESt14default_deleteIS4_ELb1EEERKNS_22PendingQueryParametersEb+0xa9) [0x79b7e0cf47c9]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext20PendingQueryInternalERNS_17ClientContextLockERKNS_10shared_ptrINS_8RelationELb1EEEb+0x184) [0x79b7e0cfa764]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb13ClientContext12PendingQueryERKNS_10shared_ptrINS_8RelationELb1EEEb+0x44) [0x79b7e0cfa954]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(+0xc3ca22) [0x79b7e043ca22]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb16DuckDBPyRelation15ExecuteInternalEb+0x14) [0x79b7e043cac4]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb16DuckDBPyRelation14ExecuteOrThrowEb+0x43) [0x79b7e043ccd3]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb16DuckDBPyRelation20ToArrowTableInternalEmb+0x1a8) [0x79b7e043d878]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(_ZN6duckdb16DuckDBPyRelation8ToPolarsEmb+0x31) [0x79b7e044e981]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(+0xcfbc8d) [0x79b7e04fbc8d]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/lib/python3.13/site-packages/_duckdb.cpython-313-x86_64-linux-gnu.so(+0xb951bf) [0x79b7e03951bf]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x15aade) [0x79b7e3d5aade]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(_PyObject_MakeTpCall+0x87) [0x79b7e3cf93d7]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(_PyEval_EvalFrameDefault+0x135d) [0x79b7e3c9217d]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(PyEval_EvalCode+0x165) [0x79b7e3e52335]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x2bc1f1) [0x79b7e3ebc1f1]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x2bc54f) [0x79b7e3ebc54f]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x2be8ff) [0x79b7e3ebe8ff]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(+0x2bee4f) [0x79b7e3ebee4f]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(Py_RunMain+0x9a9) [0x79b7e3ee9c49]
/home/henry/.pyenv/versions/3.13.7/lib/libpython3.13.so.1.0(Py_BytesMain+0x5a) [0x79b7e3eea3ba]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x79b7e3a32083]
/home/henry/.cache/uv/environments-v2/duckdb-polars-error-bd5c68c247046516/bin/python3(_start+0x2e) [0x5a33c194e09e]

This error signals an assertion failure within DuckDB. This usually occurs due to unexpected conditions or errors in the program's logic.
For more information, see https://duckdb.org/docs/stable/dev/internal_errors

OS:

Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.31

DuckDB Package Version:

1.34.0

Python Version:

3.13.7

Full Name:

Henry Harbeck

Affiliation:

None

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[paultiq]

Linking to my discussion comment here (since github doesn't link issues back to discussions): #132 (reply in thread)

If this issue is resolved, then duckdb/duckdb#13798 could be reverted which has a significant performance benefit for Polars scans.

[kylebarron]

This is expected behavior with the Arrow PyCapsule Interface; you can't use a capsule multiple times because that capsule holds a C pointer that can only be used once.

The correct behavior should be to use SELECT * from df, because that will call __arrow_c_stream__() on the Polars DataFrame producing a new capsule.

In particular, I'd say it's an anti-pattern to allow querying from the capsule itself. Users should never be able to see the capsule and should only see other Arrow objects that can produce capsules.

[paultiq]

I'm working up a proposed fix for the underlying issue in #70, which I think will solve this too, which solves the double consumption problem but not if you explicitly call arrow_c_stream

[henryharbeck]

For what its worth, the only reason I am doing this is as a workaround to not require PyArrow
ref: #132 / duckdb/duckdb#13827

(which I'm sure you're aware is a pain-point give you created the inital issue 😅)

[evertlammerts]

We're now throwing an InvalidInputException instead of an InternalException.