Arrow Pushdowns fail when *any* column is a string_view

[paultiq]

What happens?

When any column is a string_view, Arrow pushdowns fail... even if the pushdown expression doesn't involve a string_view.

Related to #187, although Vortex actually doesn't fail the below case.

To Reproduce

import pyarrow as pa
import duckdb
print(f"{duckdb.__version__=}")
arrow_table = pa.table({"id": [1], "name": ["something"]}, schema=pa.schema([("id", pa.int64()), ("name", pa.string_view())]))
print(arrow_table.schema)
duckdb.execute("SELECT * FROM arrow_table WHERE id > 90").fetchall()

duckdb.version='1.4.4.dev11'
id: int64
value: string_view
...
_duckdb.InvalidInputException: Invalid Input Error: arrow_scan: get_next failed(): NotImplemented: Function 'array_filter' has no kernel matching input types (string_view, bool)

OS:

Linux

DuckDB Package Version:

1.4.4.dev11

Python Version:

3.13

Full Name:

Paul T

Affiliation:

Iqmo

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration to reproduce the issue?

• Yes, I have

[paultiq]

This is an Arrow issue as shown below, but DuckDB could/should not attempt a pushdown if any column is a string_view (or perhaps any View).

apache/arrow#43010

import pyarrow as pa
import pyarrow.dataset as ds

table = pa.table({"id": [1, 2, 3], "value": ["a", "b", "c"]}).cast(pa.schema([("id", pa.int64()), ("value", pa.string_view())]))
scanner = ds.dataset(table).scanner(filter=ds.field("id") > 1)
scanner.to_table()

[Tishj]

This was merged upstream to fix the problem: duckdb/duckdb#20165

[paultiq]

That PR doesn't fix this issue. This issue was tested after that was merged.

That PR handles cases where the expression compares a string_view column.

This issue occurs when any column is a string_view.

Example:

import polars as pl
import duckdb
print(duckdb.__version__)

df = pl.DataFrame({"sv_col": ["abc"], "int_col": [1]})
arrow_table = df.to_arrow(compat_level=pl.CompatLevel.newest())


# Passes, because of duckdb/duckdb#20165
duckdb.execute("select * from arrow_table where sv_col = 'abc'").df()

# Still fails
duckdb.execute("select * from arrow_table where int_col = 1").df()

[paultiq]

In digging into this, and the related Vortex issue, there's a messy set of problems bigger than just this issue:

Arrow backends (Vortex, Arrow Compute, etc) support different predicates, but arrow.cpp:CanPushdown assumes coverage is universal:

• Arrow Compute's array_filter doesn't support tables that contain string_views (this issue), but Vortex DataSets do
• After Map view types when converting arrow_expressions to vortex expressions vortex-data/vortex#5765 is merged, Vortex DataSets will support equality / inequalities on string_views, but Arrow Compute will not
• Arrow Compute supports comparisons of timestamps, but Vortex DataSets do not Epic: vortex-compute compute functions vortex-data/vortex#5781. This is a new issue that raises an exception in duckdb.

In other words, duckdb makes the false assumption that all Arrow datasets support the same predicates / comparisons for pushdown.

Possible Solutions

• duckdb should be pessimistic, and deny pushdown when any backend doesn't support a particular predicate on a particular datatype
• duckdb should be optimistic, attempting to pushdown, but duckdb-python would gracefully handle an exception during pushdown - falling back to a no-pushdown case. Currently, duckdb-python will reraise the exception.
• (too complicated) duckdb would need to add rules to handle backend specific cases, handling (for instance) vortex datasets differently.
• somehow check at runtime which predicates are supported: this would involve some change in Arrow.

[danking]

FWIW, after noodling on this a bit, I've come to the conclusion that the Arrow Dataset API is pretty awkward. I suppose there's a reasonable argument that Vortex (and other Dataset implementors) should just support every Arrow expression (we could always just evaluate the expression ourselves before yielding each record batch). However, that means that duckdb will push expressions down to us that might be faster/cheaper to evaluate in duckdb. It seems like, at least for Vortex, we should push folks away from the Arrow Dataset API and towards the Vortex duckdb extension.

[paultiq]

@danking while I think I agree with your conclusion, I don't think it's related to -this- issue. This issue is arrow compute raising when array_filter is called when there's a string_view in the Arrow table.

It won't actually raise with a Vortex dataset, which makes it an opposite case where Vortex supports the pushdown where Arrow doesn't.

Just to be clear: even if users never hit the vortex dataset api, this issue still needs to be resolved for Arrow

[evertlammerts]

This PR adds a config option that gates pushdown on whether the schema contains view types. I'm not sure whether this is the way to go, feedback very welcome.