ESQL: Add times to topn status #131555

nik9000 · 2025-07-18T19:30:32Z

Adds times to the TopNOperator status, specifically the nanoseconds spent receiving the values and the nanoseconds spent emitting the values:

          {
            "operator" : "TopNOperator[count=0/1000, elementTypes=[BYTES_REF, DOUBLE], encoders=[UTF8TopNEncoder, DefaultSortable], sortOrders=[SortOrder[channel=1, asc=false, nullsFirst=true]]]",
            "status" : {
              "receive_nanos" : 192932,          <--- Page add time - expect to scale based on input
              "receive_time" : "192.9micros",    <--- Page add time as human readable string
              "emit_nanos" : 84383,              <--- Result construction time - expect to scale based on window size
              "emit_time" : "84.3micros",        <--- Result construction time as human readable string
              "occupied_rows" : 0,
              "ram_bytes_used" : 4296,
              "ram_used" : "4.1kb",
              "pages_received" : 1,
              "pages_emitted" : 1,
              "rows_received" : 1000,
              "rows_emitted" : 1000
            }

Adds times to the TopNOperator status, specifically the nanoseconds spent receiving the values and the nanoseconds spent emitting the values: ``` { "operator" : "TopNOperator[count=0/1000, elementTypes=[BYTES_REF, DOUBLE], encoders=[UTF8TopNEncoder, DefaultSortable], sortOrders=[SortOrder[channel=1, asc=false, nullsFirst=true]]]", "status" : { "receive_nanos" : 193415, <--- this row "emit_nanos" : 61, <--- and this row "occupied_rows" : 0, "ram_bytes_used" : 4296, "ram_used" : "4.1kb", "pages_received" : 1, "pages_emitted" : 1, "rows_received" : 1000, "rows_emitted" : 1000 } } ```

elasticsearchmachine · 2025-07-18T19:30:56Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

dnhatn · 2025-07-18T21:26:54Z

.../plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/topn/TopNOperator.java

@@ -548,9 +553,11 @@ public Page getOutput() {
        if (output == null || output.hasNext() == false) {
            return null;
        }
+        long start = System.nanoTime();


I think we should capture the emit time in finish() (toPages) method instead?

Looking more closely - of course. Wow. I wasn't paying attention. Thanks!

ivancea · 2025-07-21T11:44:29Z

...n/esql/compute/src/main/java/org/elasticsearch/compute/operator/topn/TopNOperatorStatus.java

+        builder.field("receive_nanos", receiveNanos);
+        builder.field("emit_nanos", emitNanos);


Other statuses also add the "X_time" with the readable time. For example:

elasticsearch/x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/HashAggregationOperator.java

Lines 436 to 439 in 59df1bf

builder.field("hash_nanos", hashNanos);

if (builder.humanReadable()) {

builder.field("hash_time", TimeValue.timeValueNanos(hashNanos));

}

Ah. Makes sense. Something I'd have asked for I think.

ivancea · 2025-07-21T11:46:42Z

.../plugin/esql/compute/src/main/java/org/elasticsearch/compute/operator/topn/TopNOperator.java

@@ -423,6 +427,7 @@ public void addInput(Page page) {
            pagesReceived++;
            rowsReceived += page.getPositionCount();
        }
+        receiveNanos += System.nanoTime() - start;


This should be in the finally, in case of exception (?)

May as well, yeah. I think the exception here is generally terminal to the process and we don't return the status, but may as well put it there so it's possible.

dnhatn

LGTM. Thanks Nik!

…king * upstream/main: (100 commits) Term vector API on stateless search nodes (elastic#129902) TEST Fix ThreadPoolMergeSchedulerStressTestIT testMergingFallsBehindAndThenCatchesUp (elastic#131636) Add inference.put_custom rest-api-spec (elastic#131660) ESQL: Fewer serverless docs in tests (elastic#131651) Skip search on indices with INDEX_REFRESH_BLOCK (elastic#129132) Mute org.elasticsearch.indices.cluster.RemoteSearchForceConnectTimeoutIT testTimeoutSetting elastic#131656 [jdk] Resolve EA OpenJDK builds to our JDK archive (elastic#131237) Add optimized path for intermediate values aggregator (elastic#131390) Correctly handling download_database_on_pipeline_creation within a pipeline processor within a default or final pipeline (elastic#131236) Refresh potential lost connections at query start for `_search` (elastic#130463) Add template_id to patterned-text type (elastic#131401) Integrate LIKE/RLIKE LIST with ReplaceStringCasingWithInsensitiveRegexMatch rule (elastic#131531) [ES|QL] Add doc for the COMPLETION command (elastic#131010) ESQL: Add times to topn status (elastic#131555) ESQL: Add asynchronous pre-optimization step for logical plan (elastic#131440) ES|QL: Improve generative tests for FORK [130015] (elastic#131206) Update index mapping update privileges (elastic#130894) ESQL: Added Sample operator NamedWritable to plugin (elastic#131541) update `kibana_system` to grant it access to `.chat-*` system index (elastic#131419) Clarify heap size configuration (elastic#131607) ...

…-tracking * upstream/main: (44 commits) Term vector API on stateless search nodes (elastic#129902) TEST Fix ThreadPoolMergeSchedulerStressTestIT testMergingFallsBehindAndThenCatchesUp (elastic#131636) Add inference.put_custom rest-api-spec (elastic#131660) ESQL: Fewer serverless docs in tests (elastic#131651) Skip search on indices with INDEX_REFRESH_BLOCK (elastic#129132) Mute org.elasticsearch.indices.cluster.RemoteSearchForceConnectTimeoutIT testTimeoutSetting elastic#131656 [jdk] Resolve EA OpenJDK builds to our JDK archive (elastic#131237) Add optimized path for intermediate values aggregator (elastic#131390) Correctly handling download_database_on_pipeline_creation within a pipeline processor within a default or final pipeline (elastic#131236) Refresh potential lost connections at query start for `_search` (elastic#130463) Add template_id to patterned-text type (elastic#131401) Integrate LIKE/RLIKE LIST with ReplaceStringCasingWithInsensitiveRegexMatch rule (elastic#131531) [ES|QL] Add doc for the COMPLETION command (elastic#131010) ESQL: Add times to topn status (elastic#131555) ESQL: Add asynchronous pre-optimization step for logical plan (elastic#131440) ES|QL: Improve generative tests for FORK [130015] (elastic#131206) Update index mapping update privileges (elastic#130894) ESQL: Added Sample operator NamedWritable to plugin (elastic#131541) update `kibana_system` to grant it access to `.chat-*` system index (elastic#131419) Clarify heap size configuration (elastic#131607) ...

nik9000 requested a review from ivancea July 18, 2025 19:30

nik9000 added >non-issue :Analytics/ES|QL AKA ESQL v9.2.0 labels Jul 18, 2025

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jul 18, 2025

dnhatn reviewed Jul 18, 2025

View reviewed changes

ivancea reviewed Jul 21, 2025

View reviewed changes

Merge branch 'main' into esql_topn_profile_time

f99aaba

nik9000 requested review from ivancea and dnhatn July 21, 2025 13:44

Merge branch 'main' into esql_topn_profile_time

8be1a6e

dnhatn approved these changes Jul 21, 2025

View reviewed changes

nik9000 merged commit b5b636a into elastic:main Jul 21, 2025
33 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ESQL: Add times to topn status #131555

ESQL: Add times to topn status #131555

Uh oh!

nik9000 commented Jul 18, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

dnhatn Jul 18, 2025

Uh oh!

nik9000 Jul 21, 2025

Uh oh!

nik9000 Jul 21, 2025

Uh oh!

ivancea Jul 21, 2025

Uh oh!

nik9000 Jul 21, 2025

Uh oh!

ivancea Jul 21, 2025

Uh oh!

nik9000 Jul 21, 2025

Uh oh!

dnhatn left a comment

Uh oh!

Uh oh!

Uh oh!

		builder.field("receive_nanos", receiveNanos);
		builder.field("emit_nanos", emitNanos);

	builder.field("hash_nanos", hashNanos);
	if (builder.humanReadable()) {
	builder.field("hash_time", TimeValue.timeValueNanos(hashNanos));
	}

ESQL: Add times to topn status #131555

ESQL: Add times to topn status #131555

Uh oh!

Conversation

nik9000 commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 18, 2025

Uh oh!

dnhatn Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

nik9000 Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

nik9000 Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

ivancea Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

nik9000 Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

ivancea Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

nik9000 Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nik9000 commented Jul 18, 2025 •

edited

Loading