Skip to content

fix: linearized operands in physical binaryexpr protobuf to avoid recursion limit#21031

Merged
Jefffrey merged 12 commits intoapache:mainfrom
haohuaijin:fix-recursion-limit
Apr 17, 2026
Merged

fix: linearized operands in physical binaryexpr protobuf to avoid recursion limit#21031
Jefffrey merged 12 commits intoapache:mainfrom
haohuaijin:fix-recursion-limit

Conversation

@haohuaijin
Copy link
Copy Markdown
Contributor

@haohuaijin haohuaijin commented Mar 18, 2026

Which issue does this PR close?

Rationale for this change

When a SQL query contains many filter conditions (e.g., 40+ AND/OR clauses in a WHERE), serializing the physical plan to protobuf and deserializing it fails with DecodeError: recursion limit reached. This is because prost has a default recursion limit of 100, and each BinaryExpr nesting consumes ~2 levels of protobuf recursion depth, so a chain of ~50 AND conditions exceeds the limit.

What changes are included in this PR?

Applied the same linearization approach that logical expressions already use that convert a left-deep tree to linearization list. Instead of encoding a chain of same-operator binary expressions as a deeply nested tree, we flatten it into a flat operands list:

Before (nested, O(n) recursion depth):

BinaryExpr(AND) { l: BinaryExpr(AND) { l: BinaryExpr(AND) { l: a, r: b }, r: c }, r: d }

After (flat, O(1) recursion depth for the chain):

BinaryExpr(AND) { operands: [a, b, c, d] }

Are these changes tested?

yes, add some test case

Are there any user-facing changes?

@github-actions github-actions bot added the proto Related to proto crate label Mar 18, 2026
Comment on lines 981 to 982
PhysicalExprNode l = 1;
PhysicalExprNode r = 2;
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we delete it or keep it for backward compatibility

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we have a stance on maintaining backward compatibility yet; so far we haven't bothered, though there is this issue raised to discuss the possibility:

I believe recently DataFusion is trying to cut down on breaking changes so perhaps it's worth to keep this backwards compatibility with that in mind

@haohuaijin
Copy link
Copy Markdown
Contributor Author

@Jefffrey @alamb would you mind taking a look?

Copy link
Copy Markdown
Contributor

@Jefffrey Jefffrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me, if we've already applied this logic to logical side and are just bringing it to physical

Comment on lines 981 to 982
PhysicalExprNode l = 1;
PhysicalExprNode r = 2;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we have a stance on maintaining backward compatibility yet; so far we haven't bothered, though there is this issue raised to discuss the possibility:

I believe recently DataFusion is trying to cut down on breaking changes so perhaps it's worth to keep this backwards compatibility with that in mind

@haohuaijin
Copy link
Copy Markdown
Contributor Author

Thanks for your reviews @Jefffrey ❤️

@Jefffrey Jefffrey added this pull request to the merge queue Apr 17, 2026
Merged via the queue into apache:main with commit e5966b5 Apr 17, 2026
31 checks passed
@Jefffrey
Copy link
Copy Markdown
Contributor

Thanks @haohuaijin

@haohuaijin haohuaijin deleted the fix-recursion-limit branch April 17, 2026 09:25
github-merge-queue bot pushed a commit that referenced this pull request Apr 17, 2026
Merging in #21031 broke main
due to #21573 landing before it
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

proto Related to proto crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants