Expose metrics for Trino Gateway to consume

[andythsu]

Problem:

Based on the Trino Contributor call last week, we want to support having a unified way of fetching cluster metrics using REST API. These cluster metrics will be used by Trino-Gateway for routing/load-balancing purposes. As a short term goal, we will focus on getting cluster-wide memory and cpu usage.

Existing solution:

Currently, metrics are scattered as mbeans, behind /metrics, and some are behind/ui/api. But they are all only available as a node-level metrics. There is no aggregated metrics available. Trino-Gateway will need to make network calls to each node in the cluster to fetch metrics, resulting in extra overhead.

Compatibility:

we want to support

• REST API access
• Adding these metrics to system catalog so they can be queried

Solution:

The coordinator will expose an API that will be used by gateway to fetch cluster metrics. Our short term goal will do it on-demand. The coordinator will only aggregate the metrics from workers when the API is called. For long term goal we may want to add caching, maintain an internal state... etc.

[ebyhr]

Reopened since we reverted the PR in #27092

[andythsu]

@findepi Let's follow up on this discussion in this issue

I appreciate the openness but I doubt that would be want we would actually want to maintain in the project.

Trino has lots of custom plugins that are maintained by others. I agree that we may not want to maintain all softwares in Trino project but since Trino Gateway and Trino work together with each other, I think it doesn't harm to have a dedicated endpoint for Trino Gateway. Similarly to UI endpoint, Trino Gateway endpoints can evolve as Trino Gateway needs evolve since Trino Gateway relies heavily on Trino

BTW trino already exports tons of metrics in standard format (that's why jmx became obsolete), so i don't quite yet understand why we need dedicated endpoints just for selected metrics useful for selected use case

If by standard format you mean /metrics then please checkout the discussion here. It should give you more context on this decision

[findepi]

If by standard format you mean /metrics then please checkout the discussion here. It should give you more context on this decision

thanks for the link!
insightful

BTW the code is back in #27101

Trino has lots of custom plugins that are maintained by others.

I wish I could share your sentiment about plugins being maintained by "others", but I cannot honestly say that I do.

If there is a test failure on a contributor's PR, it's on that contributor to fix the failure. No "others" involved.
if there is a flaky test, or the code needs update to eg work with new java version, it's always first-most the maintainers' duty to address this. I wish it was some "others" who feel responsible and do this work, just I don't see that really being the case. Not in this project and generally not in open source.

The real burden of maintaining the software project -- be it its core, or the plugins -- lies in maintainers' hand. The only exception is when there is little to no test coverage and maintainers do not need to worry, do not even need to know, that something does not work. I doubt no test coverage is the way to go though.