Commit d530276
Fix
## Which issue does this PR close?
Closes part of #18195 —
specifically the `elapsed_compute` baseline metric sub-item for Parquet
scans.
## Rationale
`EXPLAIN ANALYZE` on Parquet scans reports `elapsed_compute` values like
`14ns` for full table scans, which is misleading. The metric was never
being populated because no timer wrapped the per-batch compute work in
the Parquet scan path.
## What changes are included in this PR?
Follows the same pattern established in PR #18901 (CSV fix):
1. Added `BaselineMetrics` instantiation in `ParquetOpener::open()`
using the existing `metrics` and `partition_index` fields
2. Wrapped the per-batch stream `.map()` closure with an
`elapsed_compute` timer that measures projection, schema replacement,
and metrics copy work
Single file changed: `datafusion/datasource-parquet/src/opener.rs` (+7,
-3 lines)
## Are these changes tested?
- All 81 existing tests in `datafusion-datasource-parquet` pass
- The metric correctness is verified by observing realistic
`elapsed_compute` values in `EXPLAIN ANALYZE` output (no longer showing
nanosecond-level values for real scans)
- Per maintainer guidance from @2010YOUY01: "Testing if we have the time
measured correct is tricky, I don't think there is a good way to do it.
But for a large parquet file scan, several nanoseconds is definitely not
reasonable."
## Are there any user-facing changes?
`EXPLAIN ANALYZE` output for Parquet scans will now show accurate
`elapsed_compute` values reflecting actual CPU time spent on per-batch
processing.
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>elapsed_compute metric for Parquet DataSourceExec (#20767)1 parent f30b85c commit d530276
2 files changed
Lines changed: 9 additions & 3 deletions
File tree
- datafusion
- datasource-parquet/src
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
| 51 | + | |
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| |||
135 | 135 | | |
136 | 136 | | |
137 | 137 | | |
| 138 | + | |
138 | 139 | | |
139 | 140 | | |
140 | 141 | | |
| |||
605 | 606 | | |
606 | 607 | | |
607 | 608 | | |
| 609 | + | |
608 | 610 | | |
609 | 611 | | |
610 | 612 | | |
| |||
646 | 648 | | |
647 | 649 | | |
648 | 650 | | |
| 651 | + | |
649 | 652 | | |
650 | 653 | | |
651 | 654 | | |
| |||
671 | 674 | | |
672 | 675 | | |
673 | 676 | | |
| 677 | + | |
674 | 678 | | |
675 | | - | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
676 | 682 | | |
677 | 683 | | |
678 | 684 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
107 | | - | |
| 107 | + | |
108 | 108 | | |
109 | 109 | | |
110 | 110 | | |
| |||
0 commit comments