Skip to content

Move the spill pre-processing steps into SpillManager #18261

@2010YOUY01

Description

@2010YOUY01

Is your feature request related to a problem or challenge?

I noticed it when reviewing #18207

When writing spill files in the sort executor, it requires some pre-processing steps for the batches to spill to avoid duplicate writes, specifically

Self::organize_stringview_arrays(globally_sorted_batches)?;

This step should apply to all spilling usages, not only limited to the sort executor, so a better way is to put it inside SpillManager/InProgressSpillFile, or maybe Arrow IPC writer if possible.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions