[Questions] Understanding Quorum Queue Behavior

[stephen7921]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.2

Erlang version used

27.3.x

Operating system (distribution) used

Ubuntu 22.04.5

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

We run RabbitMQ as a 3-node on AWS service

Steps to reproduce the behavior in question

Through RabbitMQ, over 150 messages (4K bytes, multiple segment files) per second are transmitted to a single queue.
Each queue has a lifecycle that includes creation, message transmission, and deletion, and during the service, there are more than 40 active queues at any given time. (In other words, RabbitMQ is processing more than over 6,000 messages per second.)
RabbitMQ is configured in a clustered environment on AWS, and the application server exists in the same region.

When using RabbitMQ version 3.10.X with the Classic Type (without HA), no specific issues were observed in the service.
However, starting from RabbitMQ version 4.X, the default queue type is set to Quorum, and the following abnormal phenomena have been observed:
(The application does not specify the queue type during creation, so it follows the default queue type.)

Symptoms:

1. The consumer of the queue disappears, and the number of ready messages increases.
   RabbitMQ Log: missed heartbeats from client, timeout: 60s
2. Threads related to queue operations in the application server experience locks and blocks.
3. The number of file descriptors (FDs) in the application server increases.

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

While it is understood that Classic Queues are lighter in weight compared to Quorum Queues, these symptoms do not seem to be secondary effects caused by using Quorum Queues.

Could you explain why such phenomena might occur in Quorum Queues?
(If additional information is required, please let me know, but please understand that very detailed information cannot be disclosed.)

[kjnilsson]

Have you inspected the RabbitMQ server logs? That would be your first point of call.

[michaelklishin]

@stephen7921 we do not guess in this community. You haven't shared any logs or code.

You are running an outdated patch release that excludes a file descriptor leak in QQs that was fixed in 4.1.3.

The consumer of the queue disappears, and the number of ready messages increases.

This is expected because all unconfirmed deliveries are automatically requeued

Threads related to queue operations in the application server experience locks and blocks

We cannot suggest anything specific without logs from all nodes. There is a known scenario where a QQ can end up without a leader in 4.1.4 and it takes days to reproduce, that is, many won't hit it.
There is also #12366 which those using QQs as they were meant to be used

The number of file descriptors (FDs) in the application server increases

We cannot know what your "application server" does or why.

The aforementioned change in 4.1.3, rabbitmq/ra#553, is a change to RabbitMQ itself, not an "application server" or client libraries.

Opening a new connection to RabbitMQ means increasing the file (and socket) descriptor count, that's how network sockets work. I assume that can affect "application servers" (but I don't really know what exactly that means to you).