Skip to content

Commit cb694bf

Browse files
adriangbclaude
andcommitted
fix: admit zero-bytes-saved samples so skip fires for filter==projection
Q18 (and several other TPC-DS regressions) had a post-scan customer_demographics filter — `Optional(DynamicFilter)` on the single projected column `cd_demo_sk` — that burned 90 ms of CPU per scan but could never be skipped. The filter was correctly placed at PostScan (projection ⊆ filter columns ⇒ byte_ratio = 1.0 > threshold) but the mid-stream skip path never fired. Root cause: `SelectivityStats::update` only incremented `sample_count` when `batch_bytes > 0`. When the projection is a subset of the filter columns, `other_bytes_per_row = 0` and therefore `batch_bytes = 0` on every call, so the Welford counter stayed at zero, the CI upper bound stayed `None`, and the skip check short-circuited. Meanwhile the filter kept running per batch. Admit samples with `batch_bytes = 0`. The recorded effectiveness for those samples is legitimately zero (no late-materialization payoff), so the CI upper bound converges on zero after a few batches and the skip flag flips for optional filters — exactly what we want: CPU spent, no byte savings, optional ⇒ drop. Local TPC-DS sf1 (M-series, pushdown=on) vs main pushdown=off: | Query | Main(off) | Before | After | |-------|-----------|--------|-------| | Q18 | 99 | 182 | 118 | | Q67 | 312 | 503 | 346 | | Q26 | 80 | 151 | 94 | | Q85 | 149 | 246 | 157 | | Q91 | 64 | 108 | 58 | | Q53 | 103 | 144 | 99 | | Q63 | 103 | 148 | 99 | | Q13 | 399 | 558 | 376 | | Q72 | 619 | 489 | 277 | | Q24 | 379 | 70 | 70 | | Q64 | 28213 | -- | 519 | Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent fee428c commit cb694bf

1 file changed

Lines changed: 13 additions & 3 deletions

File tree

datafusion/datasource-parquet/src/selectivity.rs

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -149,10 +149,20 @@ impl SelectivityStats {
149149
self.eval_nanos += eval_nanos;
150150
self.bytes_seen += batch_bytes;
151151

152-
// Feed Welford's algorithm with per-batch effectiveness
153-
if total > 0 && eval_nanos > 0 && batch_bytes > 0 {
152+
// Feed Welford's algorithm with per-batch effectiveness. We admit
153+
// samples with `batch_bytes == 0` — that legitimately represents a
154+
// filter whose projection is a subset of its referenced columns, so
155+
// late materialization has nothing to save even when the filter
156+
// does prune rows. Recording `batch_eff = 0` for such batches lets
157+
// the mid-stream skip path detect "CPU spent, no late-
158+
// materialization payoff" and drop the filter if it is optional.
159+
if total > 0 && eval_nanos > 0 {
154160
let rows_pruned = total - matched;
155-
let bytes_per_row = batch_bytes as f64 / total as f64;
161+
let bytes_per_row = if total > 0 {
162+
batch_bytes as f64 / total as f64
163+
} else {
164+
0.0
165+
};
156166
let batch_eff =
157167
(rows_pruned as f64 * bytes_per_row) * 1e9 / eval_nanos as f64;
158168

0 commit comments

Comments
 (0)