[1.4.0] Queries don’t terminate since type change of `.arrow()`

[Dino-Burger]

What happens?

Duckdb changed the return type of arrow() from a pyarrow table to a RecordBatchReader.
The consequence of this is that some queries don’t terminate.

import duckdb

xxx = duckdb.sql("From range(5) T(a)")
yyy = xxx.arrow()

# works fine
duckdb.sql("From xxx")

# never terminates
duckdb.sql("From yyy")

# works fine
zzz = xxx.fetch_arrow_table()
duckdb.sql("From zzz")

It would be great if the query From yyy would throw an error instead of not terminating, for example

The table yyy is a RecordBatchReader which is not supported form replacement scans. Please use fetch_arrow_table() instead of fetch_record_batch() (or .arrow()).

To Reproduce

Execute above code in a venv with duckdb and pyarrow installed.

OS:

Windows

DuckDB Package Version:

1.4.0

Python Version:

3.13.2

Full Name:

Dino Burger

Affiliation:

None

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[Tishj]

That's an interesting issue 🤔
What I imagine happens is:
yyy is created, this creates a streamqueryresult, which starts an execution pipeline but blocks immediately, to be resumed when the stream is fetched from.

The scan of xxx likely kills this previous execution, starting a new one and materializing the entire result

Somehow when yyy is scanned it doesn't detect that the execution was canceled (???)
I would expect this to throw an error or produce an empty result.
But instead it seems to block indefinitely

[paultiq]

I thought this was related to something I was working on (it's not), so took a look...

You've got two threads deadlocked, with one holding the ClientContextLock and the other waiting on it.

One on ResultArrowArrayStreamWrapper::MyStreamGetNext, and the other at :ArrowArrayStreamWrapper::GetNextChunk()

 	[Inline Frame] _duckdb.cp313-win_amd64.pyd!std::_Mutex_base::lock() Line 52	C++
 	[Inline Frame] _duckdb.cp313-win_amd64.pyd!std::lock_guard<std::mutex>::{ctor}(std::mutex &) Line 445	C++
 	[Inline Frame] _duckdb.cp313-win_amd64.pyd!duckdb::ClientContextLock::{ctor}(std::mutex &) Line 322	C++
 	[Inline Frame] _duckdb.cp313-win_amd64.pyd!duckdb::make_uniq(std::mutex &) Line 65	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::ClientContext::LockContext() Line 169	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::StreamQueryResult::LockContext() Line 44	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::StreamQueryResult::IsOpen() Line 193	C++
>	_duckdb.cp313-win_amd64.pyd!duckdb::ResultArrowArrayStreamWrapper::MyStreamGetNext(ArrowArrayStream * stream, ArrowArray * out) Line 123	C++

And the other:

 	_duckdb.cp313-win_amd64.pyd!duckdb::ArrowArrayStreamWrapper::GetNextChunk() Line 56	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::ArrowTableFunction::ArrowScanParallelStateNext(duckdb::ClientContext & context, const duckdb::FunctionData * bind_data_p, duckdb::ArrowScanLocalState & state, duckdb::ArrowScanGlobalState & parallel_state) Line 124	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::ArrowTableFunction::ArrowScanInitLocalInternal(duckdb::ClientContext & context, duckdb::TableFunctionInitInput & input, duckdb::GlobalTableFunctionState * global_state_p) Line 171	C++
 	_duckdb.cp313-win_amd64.pyd!duckdb::ArrowTableFunction::ArrowScanInitLocal(duckdb::ExecutionContext & context, duckdb::TableFunctionInitInput & input, duckdb::GlobalTableFunctionState * global_state_p) Line 180	C++
>	[Inline Frame] _duckdb.cp313-win_amd64.pyd!duckdb::TableScanLocalSourceState::{ctor}(duckdb::ExecutionContext &) Line 80	C++
 	[Inline Frame] _duckdb.cp313-win_amd64.pyd!duckdb::make_uniq(duckdb::ExecutionContext &) Line 65	C++

[bodacious-bill]

This is still happening in 1.4.4. I'm starting with con = duckdb.connect() and then running sql queries with result = con.execute(sql).arrow(). Using .fetch_arrow_table() instead as a workaround appears to work so far.

It would be nice if this type change on the return value for .arrow() were documented anywhere. According to the documentation, "fetch_arrow_table" is an alias of "arrow":

https://duckdb.org/docs/stable/clients/python/conversion#apache-arrow

Is this behavior intended? Is fetch_arrow_table() no longer an alias of arrow()? Previously I just downgraded to 1.3.2 to avoid this, but I want to run 1.4.4 for the pandas 3.0.0 support.

[paultiq]

That's definitely a doc error, see #97 (comment) for a discussion of current state

[bodacious-bill]

Are the docs on github here someplace? I can update it and submit a PR

[paultiq]

https://github.com/duckdb/duckdb-web