-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Description
Describe the bug
Problem
The reduce phase on coordinator nodes was a bottleneck, causing high CPU usage and latency, especially with dedicated coordinator nodes required by customer policy.
*Solution
Optimized the reduce logic by:
* Using a merge-sort approach for key-ordered aggregations to efficiently combine buckets from multiple shards. This is useful for high-cardinality cases with keys having 2 or more duplicates ultimately resulting in less number of comparisons to perform reduce merge and fetching topN.
* Applying quickselect instead of PriorityQueue when the final size was smaller than the bucket count, avoiding heap overhead.
Changes:
rishabhmaurya@3c25140
Related component
Search:Performance
Additional Details
Plugins
NA
Screenshots
NA
Host/Environment (please complete the following information):
- OS: [e.g. iOS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Status