polars.LazyFrame don't work with from_arrow, even tough it work with raw sql query

[OutSquareCapital]

What happens?

ouput of the code example:

PS C:\Users\stett\Documents\python\pql> uv run t.py
Query (sql on LazyFrame):
WITH lf AS (SELECT * FROM arrow_scan(0x15c661746e0, 0x7ffbf06c5ab0, 0x7ffbf06c5420))SELECT * FROM lf

shape: (3, 2)
┌─────┬─────┐
│ a   ┆ b   │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1   ┆ x   │
│ 2   ┆ y   │
│ 3   ┆ z   │
└─────┴─────┘
Query (sql on DataFrame):
WITH df AS (SELECT * FROM arrow_scan(0x15c66515a90, 0x7ffbf06c5ab0, 0x7ffbf06c5420))SELECT * FROM df

shape: (3, 2)
┌─────┬─────┐
│ a   ┆ b   │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1   ┆ x   │
│ 2   ┆ y   │
│ 3   ┆ z   │
└─────┴─────┘
Query (from Arrow on DataFrame):
SELECT * FROM arrow_scan(0x15c67bb85e0, 0x7ffbf06c5ab0, 0x7ffbf06c5420) AS arrow_object_8756061e3d8cb73a

shape: (3, 2)
┌─────┬─────┐
│ a   ┆ b   │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1   ┆ x   │
│ 2   ┆ y   │
│ 3   ┆ z   │
└─────┴─────┘
Traceback (most recent call last):
  File "C:\Users\stett\Documents\python\pql\t.py", line 19, in <module>
    _qry_on(duckdb.from_arrow(lf), "from Arrow on LazyFrame")
            ~~~~~~~~~~~~~~~~~^^^^
_duckdb.InvalidInputException: Invalid Input Error: Python Object Type LazyFrame is not an accepted Arrow Object.
PS C:\Users\stett\Documents\python\pql> 

I saw that in all cases the scan_arrow function was called, and concluded that it's more likely to be a runtime type check issue rather than a real limitation with LazyFrame, hence this is why I raise this as an issue.

To Reproduce

import duckdb
import polars as pl


def _qry_on(qry: duckdb.DuckDBPyRelation, name: str) -> None:

    print(f"Query ({name}):\n{qry.sql_query()}\n")
    print(qry.pl())


if __name__ == "__main__":
    data = {"a": [1, 2, 3], "b": ["x", "y", "z"]}
    lf = pl.LazyFrame(data)
    df = pl.DataFrame(data)

    _qry_on(duckdb.sql("""SELECT * FROM lf"""), "sql on LazyFrame")
    _qry_on(duckdb.sql("""SELECT * FROM df"""), "sql on DataFrame")
    _qry_on(duckdb.from_arrow(df), "from Arrow on DataFrame")
    _qry_on(duckdb.from_arrow(lf), "from Arrow on LazyFrame")

OS:

Windows

DuckDB Package Version:

1.4.4

Python Version:

3.13.7

Full Name:

Stettler Thibaud

Affiliation:

None

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have not tested with any build

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[paultiq]

The first case select * from lf passes because replacement scans explicitly check for LazyFrames, and calls .collect() and .to_arrow() on them.

The second case fails because a LazyFrame isn't an arrow type nor does it implement the capsule interface arrow_c_stream. If you changed that to from_arrow(lf.collect()), it'd work.

Implementation Difference

The difference is because from_arrow uses the following function to determine if something is Arrow:

duckdb-python/src/duckdb_py/pyconnection.cpp

Line 2312 in 89ed9a1

PyArrowObjectType DuckDBPyConnection::GetArrowType(const py::handle &obj) {

But, replacement scans have explicit code for various types, including LazyFrames:

duckdb-python/src/duckdb_py/python_replacement_scan.cpp

Lines 149 to 150 in 694922b

	auto materialized = entry.attr("collect")();
	auto arrow_dataset = materialized.attr("to_arrow")();

Possible Improvement

The replacement scan behavior could be reimplement/refactored so that the "turn LazyFrames into Arrow" code is shared with the from_arrow path. The code seems somewhat redundant and inconsistent, as is.

I discussed this flow a bit here - #132 (reply in thread)

[cmdlineluser]

@paultiq There seems to have been some recent arrow related PRs:

• feat(python): Expose ArrowStreamExportable on python collect batches iterator pola-rs/polars#26074
• feat(python): ArrowStreamExportable and sink_delta pola-rs/polars#25994

(Not sure how relevant they are here?)

[paultiq]

Oh that is interesting: Looks like that'll add __arrow_c_stream__ to LazyFrame.

If LazyFrame implements __arrow_c_stream__ , then the from_arrow path would work. The downside is that arrow pushdowns would be bypassed (as they would with any capsule).

[OutSquareCapital]

Thank you for the answers.
It's not a critical issue anyway, more like a surprising inconsistency.

After reading #132, a tool I haven't seen mentionned is the excellent narhwals.
I would suggest taking a look at it. maybe this could be of some help if the internal conversion/check layer is moved at least partially into python (or not if I'm entirely wrong about what is done there).
And, as an unrelated side-point, would definitely be of help for the public python API -> one entry point for {data, lazy}frames instead of from_df, from_arrow, etc...

[evertlammerts]

#334 added somewhat better support for lazyframes. It seems lazyframes are not exporting streams yet though, as of the latest version (1.38.2).