change enable_unions_to_filter to false

Author: xiedeyantu

datafusion/common/src/config.rs

@@ -1354,7 +1354,7 @@ config_namespace! {
         /// with a combined filter. This optimization is conservative and only applies when the
         /// branches share the same source and compatible wrapper nodes such as identical
         /// projections or aliases.
-        pub enable_unions_to_filter: bool, default = true
+        pub enable_unions_to_filter: bool, default = false
     }
 }
 

datafusion/sqllogictest/test_files/information_schema.slt

@@ -308,7 +308,7 @@ datafusion.optimizer.enable_sort_pushdown true
 datafusion.optimizer.enable_topk_aggregation true
 datafusion.optimizer.enable_topk_dynamic_filter_pushdown true
 datafusion.optimizer.enable_topk_repartition true
-datafusion.optimizer.enable_unions_to_filter true
+datafusion.optimizer.enable_unions_to_filter false
 datafusion.optimizer.enable_window_limits true
 datafusion.optimizer.enable_window_topn false
 datafusion.optimizer.expand_views_at_output false
@@ -456,7 +456,7 @@ datafusion.optimizer.enable_sort_pushdown true Enable sort pushdown optimization
 datafusion.optimizer.enable_topk_aggregation true When set to true, the optimizer will attempt to perform limit operations during aggregations, if possible
 datafusion.optimizer.enable_topk_dynamic_filter_pushdown true When set to true, the optimizer will attempt to push down TopK dynamic filters into the file scan phase.
 datafusion.optimizer.enable_topk_repartition true When set to true, the optimizer will push TopK (Sort with fetch) below hash repartition when the partition key is a prefix of the sort key, reducing data volume before the shuffle.
-datafusion.optimizer.enable_unions_to_filter true When set to true, the logical optimizer will rewrite `UNION DISTINCT` branches that read from the same source and differ only by filter predicates into a single branch with a combined filter. This optimization is conservative and only applies when the branches share the same source and compatible wrapper nodes such as identical projections or aliases.
+datafusion.optimizer.enable_unions_to_filter false When set to true, the logical optimizer will rewrite `UNION DISTINCT` branches that read from the same source and differ only by filter predicates into a single branch with a combined filter. This optimization is conservative and only applies when the branches share the same source and compatible wrapper nodes such as identical projections or aliases.
 datafusion.optimizer.enable_window_limits true When set to true, the optimizer will attempt to push limit operations past window functions, if possible
 datafusion.optimizer.enable_window_topn false When set to true, the optimizer will replace Filter(rn<=K) → Window(ROW_NUMBER) → Sort patterns with a PartitionedTopKExec that maintains per-partition heaps, avoiding a full sort of the input. When the window partition key has low cardinality, enabling this optimization can improve performance. However, for high cardinality keys, it may cause regressions in both memory usage and runtime.
 datafusion.optimizer.expand_views_at_output false When set to true, if the returned type is a view type then the output will be coerced to a non-view. Coerces `Utf8View` to `LargeUtf8`, and `BinaryView` to `LargeBinary`.

datafusion/sqllogictest/test_files/union.slt

@@ -297,7 +297,7 @@ physical_plan
 04)--ProjectionExec: expr=[name@0 || _new as name]
 05)----DataSourceExec: partitions=1, partition_sizes=[1]
 
-# unions_to_filter is enabled by default
+# unions_to_filter is disabled by default
 
 statement ok
 set datafusion.optimizer.enable_unions_to_filter = false;

docs/source/library-user-guide/upgrading/54.0.0.md

@@ -372,3 +372,37 @@ impl Default for MyTreeNode {
     }
 }
 ```
+
+[#21075]: https://github.com/apache/datafusion/pull/21075
+
+### `UnionsToFilter` optimizer rule is now disabled by default
+
+The `datafusion.optimizer.enable_unions_to_filter` option now defaults to
+`false`. When enabled, the rule rewrites `UNION DISTINCT` branches that read the
+same source and differ only by filter predicates into a single scan with a
+combined `OR` predicate:
+
+```sql
+-- Before: two separate scans
+SELECT * FROM t WHERE a = 1
+UNION
+SELECT * FROM t WHERE a = 2
+
+-- After: one scan
+SELECT DISTINCT * FROM t WHERE a = 1 OR a = 2
+```
+
+**Who is affected:**
+
+- Queries using `UNION` against the same table with different filter
+  conditions may benefit from enabling this rule.
+
+**Migration guide:**
+
+Enable the rule when your `UNION` queries scan the same large table
+multiple times with different predicates. Avoid it when the data source handles individual equality predicates more efficiently than
+a combined `OR` (e.g., index-backed sources).
+
+```sql
+SET datafusion.optimizer.enable_unions_to_filter = true;
+```

docs/source/user-guide/configs.md

@@ -173,7 +173,7 @@ The following configuration settings are available:
 | datafusion.optimizer.expand_views_at_output                             | false                     | When set to true, if the returned type is a view type then the output will be coerced to a non-view. Coerces `Utf8View` to `LargeUtf8`, and `BinaryView` to `LargeBinary`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
 | datafusion.optimizer.enable_sort_pushdown                               | true                      | Enable sort pushdown optimization. When enabled, attempts to push sort requirements down to data sources that can natively handle them (e.g., by reversing file/row group read order). Returns **inexact ordering**: Sort operator is kept for correctness, but optimized input enables early termination for TopK queries (ORDER BY ... LIMIT N), providing significant speedup. Memory: No additional overhead (only changes read order). Future: Will add option to detect perfectly sorted data and eliminate Sort completely. Default: true                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
 | datafusion.optimizer.enable_leaf_expression_pushdown                    | true                      | When set to true, the optimizer will extract leaf expressions (such as `get_field`) from filter/sort/join nodes into projections closer to the leaf table scans, and push those projections down towards the leaf nodes.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
-| datafusion.optimizer.enable_unions_to_filter                            | true                      | When set to true, the logical optimizer will rewrite `UNION DISTINCT` branches that read from the same source and differ only by filter predicates into a single branch with a combined filter. This optimization is conservative and only applies when the branches share the same source and compatible wrapper nodes such as identical projections or aliases.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
+| datafusion.optimizer.enable_unions_to_filter                            | false                     | When set to true, the logical optimizer will rewrite `UNION DISTINCT` branches that read from the same source and differ only by filter predicates into a single branch with a combined filter. This optimization is conservative and only applies when the branches share the same source and compatible wrapper nodes such as identical projections or aliases.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
 | datafusion.explain.logical_plan_only                                    | false                     | When set to true, the explain statement will only print logical plans                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
 | datafusion.explain.physical_plan_only                                   | false                     | When set to true, the explain statement will only print physical plans                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
 | datafusion.explain.show_statistics                                      | false                     | When set to true, the explain statement will print operator statistics for physical plans                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |