Prometheus rule or metric for monitoring ra_log_wal msg queue

[ahi234]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.0

Erlang version used

27.3.x

Operating system (distribution) used

Kubernetes

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

general question

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

general question

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

general question

Steps to deploy RabbitMQ cluster

general question

Steps to reproduce the behavior in question

general question

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

I would like to monitor the message queue associated with the ra_log_wal process. In the past, I've observed that the size of this message queue tends to grow for this particular process, which can negatively impact the broker's performance and the consumption of messages due to the underlying storage limitations. To address this, I want to track the queue's status using a metric. However, I haven't been able to find any existing metric that specifically monitors the message queue for this process.

Is there an alternative approach I can implement in Grafana to monitor this queue effectively? While the message queue can be observed using the rabbitmq-diagnostics observer tool, this method isn't suitable for my specific use case.

[michaelklishin]

There is no such metric. There would have to be one such metric per Ra system (a RabbitMQ subsystem that uses Raft).

@mkuratczyk this is roughly in the domain of some of your recent work :)

[ahi234]

Thanks for the response.
Is there a plan to implement such a metric or a similar monitoring solution for the future RMQ Docker Images?

[mkuratczyk]

To be honest I thought it was already possible. The library we use to expose Prometheus metrics can expose the length of the mailbox but it doesn't seem to return anything in our case:

> curl -s http://localhost:15692/metrics/per-object | rg erlang_vm_dist_proc_message_queue_len
# TYPE erlang_vm_dist_proc_message_queue_len gauge
# HELP erlang_vm_dist_proc_message_queue_len The number of messages currently in the message queue of the process.

So it's mostly a matter of debugging why the data is not present

[ahi234]

Alright ! Thanks for sharing the curl. I will try to find an alternative way until the erlang_vm_dist_proc_message_queue_len provides data. Maybe using the uncommit log entries logic which is defined in the grafana dashboards and calculate the a rough number.