Skip to content

FileScanConfig::output_ordering must be vector of optionals #17354

@crepererum

Description

@crepererum

Describe the bug

Currently the output ordering -- which is defined per partition -- is a vector of LexOrdering:

/// All equivalent lexicographical orderings that describe the schema.
pub output_ordering: Vec<LexOrdering>,

This was kinda OK before #16217 because LexOrdering was allowed to be empty -- which was basically a sentinel value for "not ordering". Now however the code cannot specify that a partition isn't ordered anymore. This actually leads to some funky bugs like in this code here:

let Some(new_ordering) = LexOrdering::new(new_ordering) else {
continue;
};

So if the projection for a single partition leads to "unordered", then get_projected_output_ordering(input).len() < input.len(), i.e. we loose partitions. This can clearly not be right.

To Reproduce

No response

Expected behavior

FileScanConfig::output_ordering must be Vec<Option<LexOrdering>>.

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions