[Questions] Upgrade Rabbit MQ version from 3.8.9 to 3.13.7

[rahul1984p]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

other (please specify)

Erlang version used

26.0.x

Operating system (distribution) used

Ubuntu 20.04

How is RabbitMQ deployed?

Debian package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

erl -version
Erlang (SMP,ASYNC_THREADS) (BEAM) emulator version 10.7.2.3
root@staging-roger:~# rabbitmqctl status
abbitmqctl list_queues
erl -version
dpkg -l | grep erlangStatus of node rabbit@staging-roger ...
Runtime

OS PID: 636
OS: Linux
Uptime (seconds): 21590815
Is under maintenance?: false
RabbitMQ version: 3.8.9
Node name: rabbit@staging-roger
Erlang configuration: Erlang/OTP 22 [erts-10.7.2.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:64]
Erlang processes: 5407 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

• rabbitmq_delayed_message_exchange
• rabbitmq_management
• amqp_client
• rabbitmq_web_dispatch
• cowboy
• cowlib
• rabbitmq_management_agent

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@staging-roger
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@staging-roger/quorum/rabbit@staging-roger

Config files

• /etc/rabbitmq/rabbitmq.config

Log file(s)

• /var/log/rabbitmq/[email protected]
• /var/log/rabbitmq/rabbit@staging-roger_upgrade.log

Alarms

(none)

Memory

Total memory used: 0.6058 gb
Calculation strategy: rss
Memory high watermark setting: 0.8 of available memory, computed to: 13.0237 gb

allocated_unused: 0.2561 gb (40.71 %)
other_proc: 0.1017 gb (16.16 %)
queue_procs: 0.0718 gb (11.42 %)
binary: 0.0581 gb (9.24 %)
code: 0.0314 gb (4.99 %)
connection_other: 0.0292 gb (4.63 %)
other_system: 0.0209 gb (3.33 %)
plugins: 0.0197 gb (3.13 %)
mgmt_db: 0.018 gb (2.86 %)
connection_readers: 0.0074 gb (1.17 %)
other_ets: 0.0041 gb (0.66 %)
connection_channels: 0.0035 gb (0.56 %)
metrics: 0.0027 gb (0.43 %)
atom: 0.0016 gb (0.25 %)
connection_writers: 0.0012 gb (0.2 %)
msg_index: 0.0009 gb (0.15 %)
mnesia: 0.0006 gb (0.1 %)
quorum_ets: 0.0 gb (0.01 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)

File Descriptors

Total: 366, limit: 299903
Sockets: 257, limit: 269910

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 12.3798 gb

Totals

Connection count: 258
Queue count: 101
Virtual host count: 10

Listeners

Interface: [::], port: 15672, protocol: https, purpose: HTTP API over TLS (HTTPS)
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
root@staging-roger:~# dpkg -l | grep erlang
ii erlang-asn1 1:22.3.4.9-1 amd64 Erlang/OTP modules for ASN.1 support
ii erlang-base 1:22.3.4.9-1 amd64 Erlang/OTP virtual machine and base applications
ii erlang-crypto 1:22.3.4.9-1 amd64 Erlang/OTP cryptographic modules
ii erlang-eldap 1:22.3.4.9-1 amd64 Erlang/OTP LDAP library
ii erlang-inets 1:22.3.4.9-1 amd64 Erlang/OTP Internet clients and servers
ii erlang-mnesia 1:22.3.4.9-1 amd64 Erlang/OTP distributed relational/object hybrid database
ii erlang-os-mon 1:22.3.4.9-1 amd64 Erlang/OTP operating system monitor
ii erlang-parsetools 1:22.3.4.9-1 amd64 Erlang/OTP parsing tools
ii erlang-public-key 1:22.3.4.9-1 amd64 Erlang/OTP public key infrastructure
ii erlang-runtime-tools 1:22.3.4.9-1 amd64 Erlang/OTP runtime tracing/debugging tools
ii erlang-snmp 1:22.3.4.9-1 amd64 Erlang/OTP SNMP applications
ii erlang-solutions 2.0 all A package to add repository.
ii erlang-ssl 1:22.3.4.9-1 amd64 Erlang/OTP implementation of SSL
ii erlang-syntax-tools 1:22.3.4.9-1 amd64 Erlang/OTP modules for handling abstract Erlang syntax trees
ii erlang-tools 1:22.3.4.9-1 amd64 Erlang/OTP various tools
ii erlang-xmerl 1:22.3.4.9-1 amd64 Erlang/OTP XML tools

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location
cat rabbitmq.config
[
{ssl, [{versions, ['tlsv1.2']}]},
{rabbit, [
{tcp_listeners, []},
{vm_memory_high_watermark, 0.8},
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile, "/etc/ssl/xx/RapidSSLCABundle.pem"},
{certfile, "/etc/ssl/xx/sbgenomics-rapid.pem"},
{keyfile, "/etc/ssl/xx/sbgenomics-rapid.key"},
{depth, 2},
{verify, verify_peer},
{fail_if_no_peer_cert, false},
{versions, ['tlsv1.2']},
{client_renegotiation, false},
{ciphers, [
{ecdhe_ecdsa,aes_256_gcm,null,sha384},
{ecdhe_rsa,aes_256_gcm,null,sha384},
{ecdhe_ecdsa,aes_256_cbc,sha384,sha384},
{ecdhe_rsa,aes_256_cbc,sha384,sha384},
{dhe_rsa,aes_256_gcm,null,sha384},
{dhe_dss,aes_256_gcm,null,sha384},
{dhe_rsa,aes_256_cbc,sha256},
{dhe_dss,aes_256_cbc,sha256},
{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256},
{ecdhe_ecdsa,aes_128_cbc,sha256,sha256},
{ecdhe_rsa,aes_128_cbc,sha256,sha256},
{dhe_rsa,aes_128_gcm,null,sha256},
{dhe_dss,aes_128_gcm,null,sha256},
{dhe_rsa,aes_128_cbc,sha256},
{dhe_dss,aes_128_cbc,sha256},
{ecdhe_ecdsa,aes_256_cbc,sha},
{ecdhe_rsa,aes_256_cbc,sha}
]}
]}
]},

{kernel, [
{inet_default_connect_options, [{nodelay, true}]},
{inet_default_listen_options, [{nodelay, true}]}
]},

{rabbitmq_management, [
{listener, [{port, 15672},
{ip, "0.0.0.0"},
{ssl, true},
{ssl_opts, [
{cacertfile, "/etc/ssl/sbg/RapidSSLCABundle.pem"},
{certfile, "/etc/ssl/sbg/sbgenomics-rapid.pem"},
{keyfile, "/etc/ssl/sbg/sbgenomics-rapid.key"},
{versions, ['tlsv1.2']},
{client_renegotiation, false},
{ciphers, [
{ecdhe_ecdsa,aes_256_gcm,null,sha384},
{ecdhe_rsa,aes_256_gcm,null,sha384},
{ecdhe_ecdsa,aes_256_cbc,sha384,sha384},
{ecdhe_rsa,aes_256_cbc,sha384,sha384},
{dhe_rsa,aes_256_gcm,null,sha384},
{dhe_dss,aes_256_gcm,null,sha384},
{dhe_rsa,aes_256_cbc,sha256},
{dhe_dss,aes_256_cbc,sha256},
{ecdhe_ecdsa,aes_128_gcm,null,sha256},
{ecdhe_rsa,aes_128_gcm,null,sha256},
{ecdhe_ecdsa,aes_128_cbc,sha256,sha256},
{ecdhe_rsa,aes_128_cbc,sha256,sha256},
{dhe_rsa,aes_128_gcm,null,sha256},
{dhe_dss,aes_128_gcm,null,sha256},
{dhe_rsa,aes_128_cbc,sha256},
{dhe_dss,aes_128_cbc,sha256},
{ecdhe_ecdsa,aes_256_cbc,sha},
{ecdhe_rsa,aes_256_cbc,sha}
]}
]}
]}
]},

{rabbitmq_shovel, [
{shovels, [
]}
]},

{rabbitmq_stomp, [
]},

{rabbitmq_mqtt, [
]},

{rabbitmq_amqp1_0, [
]},

{rabbitmq_auth_backend_ldap, [
]}
].

Steps to deploy RabbitMQ cluster

Steps to Deploy RabbitMQ Cluster
Prepare System

Set hostname and update /etc/hosts with all cluster nodes.

Update OS packages: sudo apt update && sudo apt upgrade -y.

Install Erlang

Download Erlang .deb or .rpm compatible with RabbitMQ.

Install using: sudo dpkg -i .deb.

Install RabbitMQ

Download RabbitMQ server package.

Install using: sudo dpkg -i .deb.

Start and enable service:

bash
Copy
sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server
Synchronise Erlang Cookie Across Nodes

On node1:

bash
Copy
cat /var/lib/rabbitmq/.erlang.cookie
Copy the cookie to other nodes:

bash
Copy
scp /var/lib/rabbitmq/.erlang.cookie node2:/var/lib/rabbitmq/.erlang.cookie
scp /var/lib/rabbitmq/.erlang.cookie node3:/var/lib/rabbitmq/.erlang.cookie
Set permissions on each node:

bash
Copy
chmod 400 /var/lib/rabbitmq/.erlang.cookie
chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
Enable Management Plugin

bash
Copy
rabbitmq-plugins enable rabbitmq_management
Access via http://:15672.

Form Cluster

On node2 and node3:

bash
Copy
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit@node1
rabbitmqctl start_app
Verify Cluster Status

bash
Copy
rabbitmqctl cluster_status
Set HA Policy (Optional)

bash
Copy
rabbitmqctl set_policy ha-all ".*" '{"ha-mode":"all"}'
Test Failover

Publish test messages and stop RabbitMQ on one node to confirm queue replication and resilience.

Steps to reproduce the behavior in question

I am trying to upgrade the rabbit mq version from 3.8.9 to 3.13.7

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

Upgrade Issue: RabbitMQ Upgrade from 3.8.9 to 3.13.7

🔷 Environment Details

• Current RabbitMQ Version: 3.8.9
• Current Erlang Version: 22.3.4.3 (BEAM 10.7.2.3)
• Target RabbitMQ Version: 3.13.7
• Target Erlang Version: 26.2.5
• OS & Version: Ubuntu 20.04
• Deployment Type: Single node
• Persistence Type: Mnesia (default)

---

🔷 Upgrade Steps Planned

Need a official plan to upgrade Rabbit MQ version there.

🔷 Concerns

1. Ensuring no data loss (queues, persistent messages) during upgrades.
2. Mnesia compatibility across these jumps.
3. RabbitMQ node fails to start after restoring /var/lib/rabbitmq/mnesia from backups taken at 3.8.9.
4. Official step-by-step recommended path for upgrades from 3.8.9 to 3.13.7 with minimal downtime.

---

🔷 CLI Diagnostics Output

Please find below the output as requested:

# Output from:
rabbitmqctl status
rabbitmq-diagnostics status
rabbitmqctl list_queues
erl -version
dpkg -l | grep erlang

---

🔷 Questions for RabbitMQ Team

✅ What is the official upgrade path recommended for this version jump while preserving persistent queues and messages?

✅ Can we restore mnesia backups from older versions safely or should we only rely on definitions.json exports?

✅ Any gotchas or configuration changes between these versions we should proactively handle?

---

Thank you for your support.

[michaelklishin]

3.13.x is out of community support. You won't receive any help from the core teamwith that series unless you have an extended support license from Broadcom.

Multiple feature flags have graduated between 3.8 and 3.13. I'd use the Blue-Green Deployment upgrade option and go straight to 4.1.x.

Or you can do a series of upgrades as explained in the Release Series Upgradeability section: 3.8 to 3.9, 3.9 to 3.10, 3.11 to 3.12, 3.12 to 3.13, enabling all feature flags along the way, as the release notes explicitly instruct you to do.

Then, from 3.13.7, you can do an in-place upgrade to 4.1.x.