Relax constraint that file sort order must only reference individual columns #17419
+216
−108
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
physical-expr
pepijnve
force-pushed
the
issue_17411
branch
7 times, most recently
from
4e04412 to
4bbc81a
Compare
pepijnve
force-pushed
the
issue_17411
branch
from
49a6c8a to
5861425
Compare
…e individual columns
pepijnve
force-pushed
the
issue_17411
branch
from
5861425 to
9453640
Compare
alamb
approved these changes
Sep 10, 2025
| } | ||
| None => create_ordering(self.0.source.schema(), &self.0.order)?, | ||
| let schema = self.0.source.schema(); | ||
| let df_schema = DFSchema::try_from(Arc::clone(schema))?; |
There was a problem hiding this comment.
I wonder if (re)creating this DFSchema is necessary -- it feels like at this point we know the schema information
However, i also see we need to have a DFSchema to correctly create arbitrary PhysicalExprs so this is probably fine
There was a problem hiding this comment.
I was a bit concerned about the waste here as well, but I couldn't figure out a simple way to avoid this.
| ---- | ||
| physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/data/composite_order.csv]]}, projection=[a, b], output_ordering=[a@0 + b@1 ASC NULLS LAST], file_type=csv, has_header=true | ||
|
|
||
| # Query ordered by the declared order should be just a table scan |
| @@ -3532,7 +3532,7 @@ physical_plan | |||
| 01)BoundedWindowAggExec: wdw=[sum(multiple_ordered_table.a) ORDER BY [multiple_ordered_table.b ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Field { name: "sum(multiple_ordered_table.a) ORDER BY [multiple_ordered_table.b ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], mode=[Sorted] | |||
| 02)--CoalesceBatchesExec: target_batch_size=4096 | |||
| 03)----FilterExec: b@2 = 0 | |||
| 04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/window_2.csv]]}, projection=[a0, a, b, c, d], output_orderings=[[a@1 ASC NULLS LAST, b@2 ASC NULLS LAST], [c@3 ASC NULLS LAST]], file_type=csv, has_header=true | |||
| 04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/window_2.csv]]}, projection=[a0, a, b, c, d], output_orderings=[[c@3 ASC NULLS LAST], [a@1 ASC NULLS LAST, b@2 ASC NULLS LAST]], file_type=csv, has_header=true | |||
There was a problem hiding this comment.
do you know why the output orderings come out in a different (reverse) order now?
There was a problem hiding this comment.
No, I didn't take the time to try to understand why. That's how they're being emitted by the EquivalenceClass code. I had assumed the order was not important, but if it is I can take a closer look.
There was a problem hiding this comment.
i don't think it is important
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖 |
|
🤖: Benchmark completed Details
|
|
I am a little worried about the reported slowdowns in the sql planning benchmarks. I'll try and reproduce them locally |
|
Perhaps a two step approach would be better then where we try the “column only” version first and only use the more complex code path as fallback. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The documentation states that
WITH ORDERclauses may use non-trivial expressions. It even has an example showing the usage of this feature. In practice this does not work and the implementation is limited to simple column references.What changes are included in this PR?
physical_expr::create_lex_orderingfunction that provides a more flexible version ofphysical_expr::create_ordering.create_orderingwith its single column constraint has been retained for backwards compatibility, but should perhaps be deprecated. It does not seems possible to reimplement it in terms ofcreate_lex_orderingsince anExecutionPropsinstance is required.physical_expr::equivalence::project_orderingsconvenience function that uses the existing sort order projection logicAre these changes tested?
with ordercaseAre there any user-facing changes?
PhysicalExprinstances rather than onlyColumn.