Commit b3976d6
authored
Fix DELETE/UPDATE filter extraction when predicates are pushed down into TableScan (#19884)
## Which issue does this PR close?
* Closes #19840.
## Rationale for this change
Custom `TableProvider` implementations (notably those that support
filter pushdown) expect the DML engine to pass `WHERE` predicates into
`TableProvider::delete_from` / `TableProvider::update` so they can apply
targeted changes.
However, when the optimizer pushes predicates down into
`TableScan.filters`, the logical plan may no longer contain a top-level
`Filter` node. The existing DML planning code only collected predicates
from `Filter` nodes, so `delete_from` / `update` could receive an empty
filter list and unintentionally apply the operation to all rows.
This PR fixes filter extraction to also consider pushdown predicates
stored in `TableScan.filters`, while avoiding incorrect behavior in
multi-table contexts.
## What changes are included in this PR?
* Extend DML predicate extraction to collect filters from both:
* `LogicalPlan::Filter` nodes (splitting `AND` conjunctions), and
* `LogicalPlan::TableScan.filters` for the **target** table.
* Add target scoping for filter extraction:
* Determine the set of allowed table references for the target table
(including simple aliases via `SubqueryAlias`).
* For `Filter` nodes, retain only predicates whose column references
belong to the target table (or are unqualified).
* For `TableScan.filters`, only read filters from the target table scan
to prevent accidentally pulling predicates from source tables.
* Strip column qualifiers before passing expressions to the provider so
filters evaluate against the provider schema.
* Deduplicate extracted predicates (needed when partial/inexact pushdown
causes the same predicate to appear in both a `Filter` node and
`TableScan.filters`).
* Add comprehensive tests for DELETE/UPDATE covering:
* exact pushdown (filters live in `TableScan.filters`),
* compound predicates,
* mixed locations due to `Inexact` pushdown and per-filter decisions,
* qualifier stripping behavior.
* Temporarily reject `UPDATE ... FROM` during planning with a clear
`not_impl` error and align sqllogictest expectations.
* This avoids incorrect semantics caused by qualifier stripping /
predicate selection in multi-table update scenarios.
## Are these changes tested?
Yes.
* Added new unit tests in
`datafusion/core/tests/custom_sources_cases/dml_planning.rs` that
validate:
* pushed-down filters are extracted from `TableScan` and passed through
to providers,
* compound predicates are preserved (not over-deduped),
* mixed pushdown/residual cases still deliver the full predicate set,
* qualifiers are stripped from passed filters.
* Updated sqllogictest coverage for `UPDATE ... FROM` to assert the
feature is currently not implemented.
## Are there any user-facing changes?
* **Bug fix:** `DELETE ... WHERE ...` and `UPDATE ... WHERE ...` against
custom `TableProvider`s that use filter pushdown will now correctly pass
predicates to the provider, preventing accidental full-table operations.
* **Behavior change (planning error message / support level):** `UPDATE
... FROM` now consistently returns `This feature is not implemented:
UPDATE ... FROM is not supported` (until the underlying issue is fixed).
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed and tested.1 parent dd988f6 commit b3976d6
4 files changed
Lines changed: 610 additions & 43 deletions
File tree
- datafusion
- core
- src
- tests/custom_sources_cases
- sqllogictest/test_files
- sql/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | | - | |
| 87 | + | |
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| |||
757 | 757 | | |
758 | 758 | | |
759 | 759 | | |
760 | | - | |
| 760 | + | |
761 | 761 | | |
762 | 762 | | |
763 | 763 | | |
| |||
783 | 783 | | |
784 | 784 | | |
785 | 785 | | |
786 | | - | |
| 786 | + | |
787 | 787 | | |
788 | 788 | | |
789 | 789 | | |
| |||
2067 | 2067 | | |
2068 | 2068 | | |
2069 | 2069 | | |
2070 | | - | |
2071 | | - | |
2072 | | - | |
2073 | | - | |
2074 | 2070 | | |
2075 | | - | |
| 2071 | + | |
| 2072 | + | |
| 2073 | + | |
| 2074 | + | |
| 2075 | + | |
| 2076 | + | |
| 2077 | + | |
| 2078 | + | |
| 2079 | + | |
| 2080 | + | |
| 2081 | + | |
| 2082 | + | |
| 2083 | + | |
| 2084 | + | |
| 2085 | + | |
| 2086 | + | |
| 2087 | + | |
| 2088 | + | |
| 2089 | + | |
| 2090 | + | |
| 2091 | + | |
| 2092 | + | |
| 2093 | + | |
2076 | 2094 | | |
| 2095 | + | |
| 2096 | + | |
| 2097 | + | |
| 2098 | + | |
| 2099 | + | |
| 2100 | + | |
| 2101 | + | |
| 2102 | + | |
| 2103 | + | |
| 2104 | + | |
| 2105 | + | |
| 2106 | + | |
| 2107 | + | |
2077 | 2108 | | |
2078 | 2109 | | |
2079 | | - | |
2080 | | - | |
2081 | | - | |
| 2110 | + | |
| 2111 | + | |
| 2112 | + | |
| 2113 | + | |
| 2114 | + | |
| 2115 | + | |
| 2116 | + | |
| 2117 | + | |
| 2118 | + | |
| 2119 | + | |
| 2120 | + | |
| 2121 | + | |
| 2122 | + | |
| 2123 | + | |
| 2124 | + | |
| 2125 | + | |
| 2126 | + | |
| 2127 | + | |
| 2128 | + | |
| 2129 | + | |
| 2130 | + | |
| 2131 | + | |
| 2132 | + | |
| 2133 | + | |
| 2134 | + | |
| 2135 | + | |
| 2136 | + | |
| 2137 | + | |
| 2138 | + | |
| 2139 | + | |
| 2140 | + | |
| 2141 | + | |
| 2142 | + | |
| 2143 | + | |
| 2144 | + | |
| 2145 | + | |
| 2146 | + | |
| 2147 | + | |
| 2148 | + | |
| 2149 | + | |
| 2150 | + | |
| 2151 | + | |
| 2152 | + | |
| 2153 | + | |
| 2154 | + | |
| 2155 | + | |
| 2156 | + | |
| 2157 | + | |
| 2158 | + | |
| 2159 | + | |
| 2160 | + | |
| 2161 | + | |
| 2162 | + | |
2082 | 2163 | | |
2083 | 2164 | | |
2084 | 2165 | | |
2085 | 2166 | | |
2086 | | - | |
2087 | | - | |
| 2167 | + | |
| 2168 | + | |
| 2169 | + | |
| 2170 | + | |
| 2171 | + | |
| 2172 | + | |
| 2173 | + | |
| 2174 | + | |
| 2175 | + | |
| 2176 | + | |
| 2177 | + | |
| 2178 | + | |
| 2179 | + | |
| 2180 | + | |
| 2181 | + | |
| 2182 | + | |
| 2183 | + | |
| 2184 | + | |
| 2185 | + | |
| 2186 | + | |
| 2187 | + | |
| 2188 | + | |
| 2189 | + | |
| 2190 | + | |
| 2191 | + | |
| 2192 | + | |
| 2193 | + | |
| 2194 | + | |
| 2195 | + | |
| 2196 | + | |
| 2197 | + | |
| 2198 | + | |
| 2199 | + | |
| 2200 | + | |
| 2201 | + | |
| 2202 | + | |
| 2203 | + | |
| 2204 | + | |
| 2205 | + | |
| 2206 | + | |
| 2207 | + | |
| 2208 | + | |
| 2209 | + | |
| 2210 | + | |
| 2211 | + | |
| 2212 | + | |
2088 | 2213 | | |
2089 | 2214 | | |
2090 | 2215 | | |
| |||
0 commit comments