Skip to content

[Performance] Optimize the reduce merge at coordinator #18705

@vinaykpud

Description

@vinaykpud

Describe the bug

Problem
The reduce phase on coordinator nodes was a bottleneck, causing high CPU usage and latency, especially with dedicated coordinator nodes required by customer policy.

*Solution
Optimized the reduce logic by:
* Using a merge-sort approach for key-ordered aggregations to efficiently combine buckets from multiple shards. This is useful for high-cardinality cases with keys having 2 or more duplicates ultimately resulting in less number of comparisons to perform reduce merge and fetching topN.
* Applying quickselect instead of PriorityQueue when the final size was smaller than the bucket count, avoiding heap overhead.

Changes:
rishabhmaurya@3c25140

Related component

Search:Performance

Additional Details

Plugins
NA

Screenshots
NA

Host/Environment (please complete the following information):

  • OS: [e.g. iOS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

Type

No type

Projects

Status

🆕 New

Status

🆕 New

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions