Skip to content

Commit dab903e

Browse files
Improve memory accounting for ArrowBytesViewMap (#20077)
## Which issue does this PR close? - Closes #20074 ## Rationale for this change ArrowBytesViewMap was previously accounting for the logical number of null bits when reporting memory usage. This under-reported memory consumption is because NullBufferBuilder may allocate more memory than is currently used. Memory accounting in DataFusion is expected to reflect allocated memory rather than logical usage to ensure accurate memory tracking. ## What changes are included in this PR? - Update ArrowBytesViewMap::size to use NullBufferBuilder::allocated_size instead of calculating size from the number of used null bits. ## Are these changes tested? - Yes. Existing tests were run: - cargo test -p datafusion-physical-expr-common ## Are there any user-facing changes? - No. This change only affects internal memory accounting and does not alter query behavior or public APIs. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 030621c commit dab903e

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

datafusion/physical-expr-common/src/binary_view_map.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -437,7 +437,7 @@ where
437437
let views_size = self.views.len() * size_of::<u128>();
438438
let in_progress_size = self.in_progress.capacity();
439439
let completed_size: usize = self.completed.iter().map(|b| b.len()).sum();
440-
let nulls_size = self.nulls.len() / 8;
440+
let nulls_size = self.nulls.allocated_size();
441441

442442
self.map_size
443443
+ views_size

0 commit comments

Comments
 (0)