[Questions] Consumers of quorum queue repeatedly receive messages #14824

dormanze · 2025-10-27T09:37:58Z

dormanze
Oct 27, 2025

Community Support Policy

I have read RabbitMQ's Community Support Policy
I run RabbitMQ 4.x, the only series currently covered by community support
I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.6

Erlang version used

26.2.x

Operating system (distribution) used

linux

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

config rabbitmq and start

Steps to reproduce the behavior in question

I don't know

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

quest 1:
I have 2 consumers but UI show only one consumer.

quest 2:
I pushed a message, and occasionally the same message was sent to both of my consumers at the same time.
trace message(my message have unique message ID, such as :8d5c5105-b313-11f0-a614-68f400000698):

================================================================================
2025-10-27T17:01:40.597+08:00: Message published

Node:         rabbit@rabbitmqservice-2
Connection:   [3fff:75:2006::210:2cd]:60810 -> [3fff:75:2006::210:1ef]:5673
Virtual host: /
User:         rabbitmq_admin
Channel:      2
Exchange:     mateinfo.durable.quorum.queues
Routing keys: [<<"1002.asyn-task-queue-adc-batch">>]
Routed queues: [<<"mateinfo.1002.asyn-task-queue-adc-batch-quorum">>]
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,
                [{<<"X-B3-ParentSpanId">>,longstr,<<"d7652dd638406a17">>},
                 {<<"X-B3-Sampled">>,longstr,<<"0">>},
                 {<<"X-B3-SpanId">>,longstr,<<"95e78eb8216e48b9">>},
                 {<<"X-B3-TraceId">>,longstr,<<"b0d9f3dd0c9712be">>},
                 {<<"come_from">>,longstr,
                  <<"adc-schedule-job-adc-schedule-job-5b484ff776-zkwmb-fc18:0:0:0:0:0:0:698">>},
                 {<<"runtimeLanguage">>,void,undefined},
                 {<<"runtimeTimeZone">>,void,undefined},
                 {<<"tenant_id">>,longstr,<<"1002">>},
                 {<<"timestamp">>,long,1761555700595},
                 {<<"x-gde-call-chain-depth">>,longstr,<<"1">>},
                 {<<"x-gde-locale">>,longstr,<<"zh_CN">>},
                 {<<"x-gde-src-app">>,void,undefined},
                 {<<"x-gde-src-app-for-transfer">>,void,undefined},
                 {<<"x-gde-src-app-invoke">>,void,undefined},
                 {<<"x-gde-src-business-field">>,void,undefined},
                 {<<"x-gde-src-item">>,void,undefined},
                 {<<"x-gde-src-item-type">>,void,undefined},
                 {<<"x-gde-src-js-invoke">>,void,undefined},
                 {<<"x-gde-src-module">>,void,undefined},
                 {<<"x-gde-src-module-legacy">>,void,undefined},
                 {<<"x-gde-src-node">>,longstr,<<"ows-pm-middleware02">>},
                 {<<"x-gde-src-service">>,longstr,<<"adc-schedule-job">>},
                 {<<"x-gde-src-user-address">>,longstr,<<"75.254.194.136">>},
                 {<<"x-gde-src-user-agent">>,longstr,
                  <<"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0">>},
                 {<<"x-gde-src-user-name">>,longstr,<<"admin">>},
                 {<<"x-gde-tenant-id">>,longstr,<<"1002">>},
                 {<<"x-gde-username">>,longstr,<<"admin">>}]},
               {<<"content_type">>,longstr,<<"text/plain">>}]
Payload: 
{"action":"Creat","taskDetail":{"durable":true,"task_id":"8d5c5105-b313-11f0-a614-68f400000698","task_type":"import-task","target_service":"adc-batch","tenant_id":"1002","user":null,"task_status":"Waiting","is_durable":true,"timeout_seconds":7200,"priority":1,"task_input":"{\"type\":\"excel\",\"source\":\"WEB_REST\",\"output\":null,\"uuid\":null,\"subTaskType\":null,\"templates\":[\"test/test/test_jskd_import\"],\"progressIds\":[\"test/test/test_jskd_import\"],\"user\":\"admin\",\"callback\":null,\"asyncTaskRunningContext\":null,\"removeFile\":false,\"fileToken\":\"eyJleHBpcmVUaW1lIjoxNzYxNjQyMTAwNTgxLCJ0ZW1wRmlsZURpciI6Ii90ZW1wL2FkYy1iYXRjaC8yMDI1LTEwLTI3LTA5LTAxLTQwLzVkNjFhZDg5LWI4MDYtNGZlYy1iYjIxLWJlZjhjNmI3YzY0OSIsImZpbGVEaXIiOiJhZGMtYmF0Y2gvaW1wb3J0LXRhc2stZmlsZS8xNzYxNTU1NzAwNTE1LzhkNTE0NzhkLWIzMTMtMTFmMC05ZTUzLTY4ZjQwMDAwMDZlNCIsInN0YXRlIjoiRVhJU1RJTkciLCJmaWxlQ2hlY2tSdWxlIjp7Im1heEZpbGVDb3VudCI6NSwibWF4RmlsZVNpemUiOjUyNDI4ODAwLCJwZXJtaXNzaW9uIjoiQ1JVRCJ9fUBAQDMwMzAzMzUzODYxMzdiNDhjYjlhNTU1MGZlYmNkNTg0MWFhZGQwYmZhZGYwOGQ1MTBlZTFlNjdhNWFmYjU1ZTk0NGY5Nzc=\",\"fileName\":\"test_jskd_import.xlsx\",\"dirPath\":null,\"batchImport\":false,\"batchImportSize\":1000,\"parameterNode\":null,\"userAddress\":\"75.254.194.136\",\"timeOut\":false,\"truncateModel\":false,\"inMemory\":false,\"dataImportContextList\":[],\"parameters\":{\"from\":\"adc-studio-web\",\"locale\":\"zh_CN\"},\"syncImport\":false}","remark":null,"task_distribute_type":"Queue","create_time":1761555700582,"create_from":"adc-batch-76f56cd448-vzrjn-fc18:0:0:0:0:0:0:6e4","callback_service_uri":null,"callback_parameters":null,"callback_service_method":null}}

================================================================================
2025-10-27T17:01:40.600+08:00: Message received

Node:         rabbit@rabbitmqservice-1
Connection:   [3fff:75:2006::210:312]:54094 -> [3fff:75:2006::210:2cd]:5673
Virtual host: /
User:         rabbitmq_admin
Channel:      12
Exchange:     mateinfo.durable.quorum.queues
Routing keys: [<<"1002.asyn-task-queue-adc-batch">>]
Queue:        mateinfo.1002.asyn-task-queue-adc-batch-quorum
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,
                [{<<"X-B3-ParentSpanId">>,longstr,<<"d7652dd638406a17">>},
                 {<<"X-B3-Sampled">>,longstr,<<"0">>},
                 {<<"X-B3-SpanId">>,longstr,<<"95e78eb8216e48b9">>},
                 {<<"X-B3-TraceId">>,longstr,<<"b0d9f3dd0c9712be">>},
                 {<<"come_from">>,longstr,
                  <<"adc-schedule-job-adc-schedule-job-5b484ff776-zkwmb-fc18:0:0:0:0:0:0:698">>},
                 {<<"runtimeLanguage">>,void,undefined},
                 {<<"runtimeTimeZone">>,void,undefined},
                 {<<"tenant_id">>,longstr,<<"1002">>},
                 {<<"timestamp">>,long,1761555700595},
                 {<<"x-gde-call-chain-depth">>,longstr,<<"1">>},
                 {<<"x-gde-locale">>,longstr,<<"zh_CN">>},
                 {<<"x-gde-src-app">>,void,undefined},
                 {<<"x-gde-src-app-for-transfer">>,void,undefined},
                 {<<"x-gde-src-app-invoke">>,void,undefined},
                 {<<"x-gde-src-business-field">>,void,undefined},
                 {<<"x-gde-src-item">>,void,undefined},
                 {<<"x-gde-src-item-type">>,void,undefined},
                 {<<"x-gde-src-js-invoke">>,void,undefined},
                 {<<"x-gde-src-module">>,void,undefined},
                 {<<"x-gde-src-module-legacy">>,void,undefined},
                 {<<"x-gde-src-node">>,longstr,<<"ows-pm-middleware02">>},
                 {<<"x-gde-src-service">>,longstr,<<"adc-schedule-job">>},
                 {<<"x-gde-src-user-address">>,longstr,<<"75.254.194.136">>},
                 {<<"x-gde-src-user-agent">>,longstr,
                  <<"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0">>},
                 {<<"x-gde-src-user-name">>,longstr,<<"admin">>},
                 {<<"x-gde-tenant-id">>,longstr,<<"1002">>},
                 {<<"x-gde-username">>,longstr,<<"admin">>}]},
               {<<"content_type">>,longstr,<<"text/plain">>}]
Payload: 
{"action":"Creat","taskDetail":{"durable":true,"task_id":"8d5c5105-b313-11f0-a614-68f400000698","task_type":"import-task","target_service":"adc-batch","tenant_id":"1002","user":null,"task_status":"Waiting","is_durable":true,"timeout_seconds":7200,"priority":1,"task_input":"{\"type\":\"excel\",\"source\":\"WEB_REST\",\"output\":null,\"uuid\":null,\"subTaskType\":null,\"templates\":[\"test/test/test_jskd_import\"],\"progressIds\":[\"test/test/test_jskd_import\"],\"user\":\"admin\",\"callback\":null,\"asyncTaskRunningContext\":null,\"removeFile\":false,\"fileToken\":\"eyJleHBpcmVUaW1lIjoxNzYxNjQyMTAwNTgxLCJ0ZW1wRmlsZURpciI6Ii90ZW1wL2FkYy1iYXRjaC8yMDI1LTEwLTI3LTA5LTAxLTQwLzVkNjFhZDg5LWI4MDYtNGZlYy1iYjIxLWJlZjhjNmI3YzY0OSIsImZpbGVEaXIiOiJhZGMtYmF0Y2gvaW1wb3J0LXRhc2stZmlsZS8xNzYxNTU1NzAwNTE1LzhkNTE0NzhkLWIzMTMtMTFmMC05ZTUzLTY4ZjQwMDAwMDZlNCIsInN0YXRlIjoiRVhJU1RJTkciLCJmaWxlQ2hlY2tSdWxlIjp7Im1heEZpbGVDb3VudCI6NSwibWF4RmlsZVNpemUiOjUyNDI4ODAwLCJwZXJtaXNzaW9uIjoiQ1JVRCJ9fUBAQDMwMzAzMzUzODYxMzdiNDhjYjlhNTU1MGZlYmNkNTg0MWFhZGQwYmZhZGYwOGQ1MTBlZTFlNjdhNWFmYjU1ZTk0NGY5Nzc=\",\"fileName\":\"test_jskd_import.xlsx\",\"dirPath\":null,\"batchImport\":false,\"batchImportSize\":1000,\"parameterNode\":null,\"userAddress\":\"75.254.194.136\",\"timeOut\":false,\"truncateModel\":false,\"inMemory\":false,\"dataImportContextList\":[],\"parameters\":{\"from\":\"adc-studio-web\",\"locale\":\"zh_CN\"},\"syncImport\":false}","remark":null,"task_distribute_type":"Queue","create_time":1761555700582,"create_from":"adc-batch-76f56cd448-vzrjn-fc18:0:0:0:0:0:0:6e4","callback_service_uri":null,"callback_parameters":null,"callback_service_method":null}}

================================================================================
2025-10-27T17:01:40.601+08:00: Message received

Node:         rabbit@rabbitmqservice-2
Connection:   [3fff:75:2006::210:380]:53044 -> [3fff:75:2006::210:1ef]:5673
Virtual host: /
User:         rabbitmq_admin
Channel:      12
Exchange:     mateinfo.durable.quorum.queues
Routing keys: [<<"1002.asyn-task-queue-adc-batch">>]
Queue:        mateinfo.1002.asyn-task-queue-adc-batch-quorum
Properties:   [{<<"priority">>,signedint,0},
               {<<"delivery_mode">>,signedint,2},
               {<<"headers">>,table,
                [{<<"X-B3-ParentSpanId">>,longstr,<<"d7652dd638406a17">>},
                 {<<"X-B3-Sampled">>,longstr,<<"0">>},
                 {<<"X-B3-SpanId">>,longstr,<<"95e78eb8216e48b9">>},
                 {<<"X-B3-TraceId">>,longstr,<<"b0d9f3dd0c9712be">>},
                 {<<"come_from">>,longstr,
                  <<"adc-schedule-job-adc-schedule-job-5b484ff776-zkwmb-fc18:0:0:0:0:0:0:698">>},
                 {<<"runtimeLanguage">>,void,undefined},
                 {<<"runtimeTimeZone">>,void,undefined},
                 {<<"tenant_id">>,longstr,<<"1002">>},
                 {<<"timestamp">>,long,1761555700595},
                 {<<"x-gde-call-chain-depth">>,longstr,<<"1">>},
                 {<<"x-gde-locale">>,longstr,<<"zh_CN">>},
                 {<<"x-gde-src-app">>,void,undefined},
                 {<<"x-gde-src-app-for-transfer">>,void,undefined},
                 {<<"x-gde-src-app-invoke">>,void,undefined},
                 {<<"x-gde-src-business-field">>,void,undefined},
                 {<<"x-gde-src-item">>,void,undefined},
                 {<<"x-gde-src-item-type">>,void,undefined},
                 {<<"x-gde-src-js-invoke">>,void,undefined},
                 {<<"x-gde-src-module">>,void,undefined},
                 {<<"x-gde-src-module-legacy">>,void,undefined},
                 {<<"x-gde-src-node">>,longstr,<<"ows-pm-middleware02">>},
                 {<<"x-gde-src-service">>,longstr,<<"adc-schedule-job">>},
                 {<<"x-gde-src-user-address">>,longstr,<<"75.254.194.136">>},
                 {<<"x-gde-src-user-agent">>,longstr,
                  <<"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0">>},
                 {<<"x-gde-src-user-name">>,longstr,<<"admin">>},
                 {<<"x-gde-tenant-id">>,longstr,<<"1002">>},
                 {<<"x-gde-username">>,longstr,<<"admin">>}]},
               {<<"content_type">>,longstr,<<"text/plain">>}]
Payload: 
{"action":"Creat","taskDetail":{"durable":true,"task_id":"8d5c5105-b313-11f0-a614-68f400000698","task_type":"import-task","target_service":"adc-batch","tenant_id":"1002","user":null,"task_status":"Waiting","is_durable":true,"timeout_seconds":7200,"priority":1,"task_input":"{\"type\":\"excel\",\"source\":\"WEB_REST\",\"output\":null,\"uuid\":null,\"subTaskType\":null,\"templates\":[\"test/test/test_jskd_import\"],\"progressIds\":[\"test/test/test_jskd_import\"],\"user\":\"admin\",\"callback\":null,\"asyncTaskRunningContext\":null,\"removeFile\":false,\"fileToken\":\"eyJleHBpcmVUaW1lIjoxNzYxNjQyMTAwNTgxLCJ0ZW1wRmlsZURpciI6Ii90ZW1wL2FkYy1iYXRjaC8yMDI1LTEwLTI3LTA5LTAxLTQwLzVkNjFhZDg5LWI4MDYtNGZlYy1iYjIxLWJlZjhjNmI3YzY0OSIsImZpbGVEaXIiOiJhZGMtYmF0Y2gvaW1wb3J0LXRhc2stZmlsZS8xNzYxNTU1NzAwNTE1LzhkNTE0NzhkLWIzMTMtMTFmMC05ZTUzLTY4ZjQwMDAwMDZlNCIsInN0YXRlIjoiRVhJU1RJTkciLCJmaWxlQ2hlY2tSdWxlIjp7Im1heEZpbGVDb3VudCI6NSwibWF4RmlsZVNpemUiOjUyNDI4ODAwLCJwZXJtaXNzaW9uIjoiQ1JVRCJ9fUBAQDMwMzAzMzUzODYxMzdiNDhjYjlhNTU1MGZlYmNkNTg0MWFhZGQwYmZhZGYwOGQ1MTBlZTFlNjdhNWFmYjU1ZTk0NGY5Nzc=\",\"fileName\":\"test_jskd_import.xlsx\",\"dirPath\":null,\"batchImport\":false,\"batchImportSize\":1000,\"parameterNode\":null,\"userAddress\":\"75.254.194.136\",\"timeOut\":false,\"truncateModel\":false,\"inMemory\":false,\"dataImportContextList\":[],\"parameters\":{\"from\":\"adc-studio-web\",\"locale\":\"zh_CN\"},\"syncImport\":false}","remark":null,"task_distribute_type":"Queue","create_time":1761555700582,"create_from":"adc-batch-76f56cd448-vzrjn-fc18:0:0:0:0:0:0:6e4","callback_service_uri":null,"callback_parameters":null,"callback_service_method":null}}

Answered by dormanze

Oct 27, 2025

I have checked the historical metrics of the nodes and found that the disk of one RabbitMQ node was once full. I'm not sure if this is the reason for the queue anomaly.

View full answer

dormanze · 2025-10-27T11:43:00Z

dormanze
Oct 27, 2025
Author

After I restarted all my consumer clients, a more strange situation occurred: the consumers were listening normally, but all messages were in the Unacked state, yet my consumers did not receive any messages.

0 replies

dormanze · 2025-10-27T11:55:08Z

dormanze
Oct 27, 2025
Author

I have checked the historical metrics of the nodes and found that the disk of one RabbitMQ node was once full. I'm not sure if this is the reason for the queue anomaly.

1 reply

michaelklishin Oct 27, 2025
Maintainer

If a node is out of disk space, you never know what it could not persist to the Raft log.

The recommendation is to decomission and replace such nodes, and have at the very least OS-level monitoring in place.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Questions] Consumers of quorum queue repeatedly receive messages #14824

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 1 reply

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

[Questions] Consumers of quorum queue repeatedly receive messages #14824

Uh oh!

dormanze Oct 27, 2025

Community Support Policy

RabbitMQ version used

Erlang version used

Operating system (distribution) used

How is RabbitMQ deployed?

rabbitmq-diagnostics status output

Logs from node 1 (with sensitive values edited out)

Logs from node 2 (if applicable, with sensitive values edited out)

Logs from node 3 (if applicable, with sensitive values edited out)

rabbitmq.conf

Steps to deploy RabbitMQ cluster

Steps to reproduce the behavior in question

advanced.config

Application code

Kubernetes deployment file

What problem are you trying to solve?

Replies: 2 comments · 1 reply

Uh oh!

dormanze Oct 27, 2025 Author

Uh oh!

dormanze Oct 27, 2025 Author

Uh oh!

michaelklishin Oct 27, 2025 Maintainer

dormanze
Oct 27, 2025

Replies: 2 comments 1 reply

dormanze
Oct 27, 2025
Author

dormanze
Oct 27, 2025
Author

michaelklishin Oct 27, 2025
Maintainer