Node cannot join existing cluster

[hac3ru]

Describe the bug

I had a 3 node cluster. Had to reinstall one of the VMs, I ran rabbitmqctl stop_app, I ran the cluster_forget_node on the remaining nodes and reinstalled the VM. Same hostname, same IP address, everything.
I reinstalled RabbitMQ on the machine, copied the erlang cookie, copied the config file, started the service and ran

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit@rabbitmq-01

but I'm getting:

2025-10-09 16:53:55.755705+02:00 [notice] <0.7042.0> RabbitMQ metadata store: candidate -> leader in term: 4 machine version: 1, last applied 4
2025-10-09 16:53:55.765637+02:00 [notice] <0.7005.0> Feature flags: checking nodes `rabbit@rabbitmq-02` and `rabbit@rabbitmq-01` compatibility...
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags: error while running:
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:   rabbit_ff_registry_wrapper:inventory[]
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags: on node `rabbit@rabbitmq-03`:
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:   exception error: {erpc,timeout}
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:     in function  erpc:call/5 (erpc.erl, line 1380)
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:     in call from rabbit_ff_controller:rpc_call/5 (src/rabbit_ff_controller.erl, line 1613)
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0> Feature flags:     in call from rabbit_ff_controller:'-rpc_calls/5-fun-1-'/7 (src/rabbit_ff_controller.erl, line 1645)
2025-10-09 16:54:55.767482+02:00 [error] <0.7048.0>
2025-10-09 16:54:55.767714+02:00 [warning] <0.7005.0> Feature flags: error while collecting inventory from nodes ['rabbit@rabbitmq-01','rabbit@rabbitmq-02','rabbit@rabbitmq-03']:
2025-10-09 16:54:55.767714+02:00 [warning] <0.7005.0> {error,{erpc,timeout}}
2025-10-09 16:54:57.617301+02:00 [error] <0.6906.0> ** Node 'rabbit@rabbitmq-03' not responding **
2025-10-09 16:54:57.617301+02:00 [error] <0.6906.0> ** Removing (timedout) connection **
2025-10-09 16:54:57.617301+02:00 [error] <0.6906.0>
2025-10-09 16:54:57.637616+02:00 [info] <0.7005.0>
2025-10-09 16:54:57.637616+02:00 [info] <0.7005.0>  Starting RabbitMQ 4.1.4 on Erlang 27.3.4.2 [jit]
2025-10-09 16:54:57.637616+02:00 [info] <0.7005.0>  Copyright (c) 2007-2025 Broadcom Inc and/or its subsidiaries
2025-10-09 16:54:57.637616+02:00 [info] <0.7005.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  node           : rabbit@rabbitmq-02
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  home dir       : /var/lib/rabbitmq
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  config file(s) : /etc/rabbitmq/rabbitmq.conf
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  cookie hash    : UvHVskhv44N7QwUVknDo1w==
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  log(s)         : /var/log/rabbitmq/rabbit.log
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>                 : <stdout>
2025-10-09 16:54:57.638865+02:00 [info] <0.7005.0>  data dir       : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-02
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags: error while running:
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:   rabbit_ff_registry_wrapper:inventory[]
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags: on node `rabbit@rabbitmq-03`:
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:   exception error: {erpc,timeout}
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:     in function  erpc:call/5 (erpc.erl, line 1380)
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:     in call from rabbit_ff_controller:rpc_call/5 (src/rabbit_ff_controller.erl, line 1613)
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0> Feature flags:     in call from rabbit_ff_controller:'-rpc_calls/5-fun-1-'/7 (src/rabbit_ff_controller.erl, line 1645)
2025-10-09 16:55:58.857372+02:00 [error] <0.7228.0>
2025-10-09 16:56:12.622288+02:00 [error] <0.7111.0> ** Node 'rabbit@rabbitmq-03' not responding **
2025-10-09 16:56:12.622288+02:00 [error] <0.7111.0> ** Removing (timedout) connection **
2025-10-09 16:56:12.622288+02:00 [error] <0.7111.0>
2025-10-09 16:56:12.626374+02:00 [error] <0.7093.0> Mnesia('rabbit@rabbitmq-02'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@rabbitmq-03'}
2025-10-09 16:56:12.626374+02:00 [error] <0.7093.0>
2025-10-09 16:56:12.629389+02:00 [notice] <0.45.0> Application mnesia exited with reason: stopped
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0>
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0> BOOT FAILED
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0> ===========
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0> Error during startup: {error,{erpc,timeout}}
2025-10-09 16:56:12.629529+02:00 [error] <0.7005.0>
2025-10-09 16:56:13.630349+02:00 [notice] <0.45.0> Application rabbit exited with reason: {{erpc,timeout},{rabbit,start,[normal,[]]}}
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>   crasher:
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     initial call: application_master:init/3
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     pid: <0.7004.0>
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     registered_name: []
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     exception exit: {{erpc,timeout},{rabbit,start,[normal,[]]}}
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>       in function  application_master:init/3 (application_master.erl, line 143)
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     ancestors: [application_controller,<0.10.0>]
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     message_queue_len: 1
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     messages: [{'EXIT',<0.7005.0>,normal}]
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     links: [<0.45.0>]
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     dictionary: []
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     trap_exit: true
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     status: running
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     heap_size: 2586
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     stack_size: 29
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>     reductions: 76
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>   neighbours:
2025-10-09 16:56:13.630409+02:00 [error] <0.7004.0>
2025-10-09 16:56:13.633831+02:00 [notice] <0.45.0> Application khepri exited with reason: stopped
2025-10-09 16:56:13.636447+02:00 [notice] <0.45.0> Application sysmon_handler exited with reason: stopped
2025-10-09 16:56:13.638643+02:00 [notice] <0.45.0> Application ra exited with reason: stopped
2025-10-09 16:56:13.639827+02:00 [notice] <0.45.0> Application os_mon exited with reason: stopped
2025-10-09 16:56:13.641014+02:00 [notice] <0.45.0> Application rabbitmq_prelaunch exited with reason: stopped

Somehow, it says that node3 is unresponsive, even though 1 and 3 are clustered and working. If I stop node 3, and try to join 2, it works. 1 and 2 can form the cluster, but the second I try to bring back node 3, I get the same error as above, but it's node2 who timesout.
What's going on?

Reproduction steps

1. Install a RabbitMQ cluster on 3 nodes (Ubuntu 24.04) via the RabbitMQ repos
2. Form a cluster - I used a config file to do it, I've only set the cluster name and the nodes.
3. Make sure all is healthy.
4. Remove a node from the cluster
5. Reinstall the OS on that node - and reinstall RabbitMQ. Give it the same name, same IP, etc.
6. Try to join the cluster on this node.

I was able to reproduce this three times already. As additional info, I'm running RabbitMQ 4.1.4-1

Expected behavior

The node should join the cluster, as any new node would.

Additional context

No response

[lukebakken]

I converted this to a discussion because there is no evidence this is a bug in RabbitMQ.

This is a decent start to a report, but what is missing are the exact commands, and transcripts of you running those commands, when you remove and reinstall RabbitMQ. There is a lot of information missing that you're asking us to guess.

In steps 4 and 5, what happens if you remove the node from the cluster, and uninstall RabbitMQ from that node (remove data directory, configuration). Then, re-install RabbitMQ and attempt to re-join the cluster. If that succeeds, then there must be something in how you're re-building the node that isn't quite identical to before.

[lukebakken]

Also, the weird thing is that if I stop one of the nodes, node 3 can join

Yes, that is weird. If I can find the time, I'll try to reproduce this using docker.

If you could perform the steps that I asked, it would shine some light on this:

• Stop RabbitMQ on a node.
• Uninstall RabbitMQ and delete everything associated with it on that node.
• Use forget_cluster_node on another node to forget the one you just removed.
• Re-install, re-apply configuration, and re-start RabbitMQ on that node. It should re-join the cluster based on your configuration file.

[michaelklishin]

@hac3ru if a VM is deleted, chances are this is one of the two classic scenarios where a hostname changes between node starts or the data directory is removed.

In both cases the node can end up generating a different Erlang cookie (shared secret) locally, which means it won't be able to join a cluster that uses a different cookie, by design.

There is a distinct error message that will be logged in such cases by the existing cluster members such reset node tries to contact.

In case it is not obvious why a hostname change can have such an effect: by default, RabbitMQ node data directory path includes a hostname, for historical reasons and to make it possible to run multi-node integration tests on a single hosts easier.

[hac3ru]

Hello,

I am aware of the /var/lib/rabbitmq/.erlang_cookie file, and that has been copied over from one of the other nodes, and I ran chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang_cookie and chmod 0400 /var/lib/rabbitmq/.erlang_cookie
Indeed, the first time I reinstalled hte node, I got a message saying
Connection attempt from node 'rabbit-1939-2495@rabbitmq-02' rejected. Invalid challenge reply.

So the new node has nothing in /var/lib/rabbitmq/mnesia/ and the .erlang_cookie matches.

[michaelklishin]

@hac3ru sorry but

Connection attempt from node 'rabbit-1939-2495@rabbitmq-02' rejected. Invalid challenge reply.

is very specific: the Erlang cookie does not match. This message is logged in only a few very closely related scenarios.

One known scenario where this can happen is RabbitMQ CLI tools running before the cookie file is created by the node. Then the CLI tools will generate a random cookie that won't match the one used by the existing cluster.

[hac3ru]

This is so fun. I'm really trying to help by pointing out a freak scenario but it seems that all you wanna do is close this thread. Maybe if reading what people are actually writing in here, it's go way better.
I said that indeed, the first time I reinstalled the node I got the erlang cookie error. That is now fixed, as I deleted the entire /var/lib/rabbitmq folder, recreated it, copied the cookie (made sure it's 100% the same by comparing the md5sum hash) and now THAT IS NOT THE PROBLEM!

I'll say it again: That error did happen on the first reinstall, on the first start of the RabbitMQ service, because I FORGOT to copy the cookie. It's not the case anymore.

@lukebakken unfortunately, I'm not able to reproduce it anymore. I have done followed the exact same steps and I'm unable to replicate it anymore....
Just in case it helps someone - summary: I had a 3 node RabbitMQ cluster - healthy and all. I halted one of the VMs, removed it from the cluster, deleted the VM. I have provisioned a new VM, using the same hostname and the same IP, and was getting the error

2025-10-09 16:54:55.767714+02:00 [warning] <0.7005.0> Feature flags: error while collecting inventory from nodes ['rabbit@rabbitmq-01','rabbit@rabbitmq-02','rabbit@rabbitmq-03']:
2025-10-09 16:54:55.767714+02:00 [warning] <0.7005.0> {error,{erpc,timeout}}
2025-10-09 16:54:57.617301+02:00 [error] <0.6906.0> ** Node 'rabbit@rabbitmq-03' not responding **
2025-10-09 16:54:57.617301+02:00 [error] <0.6906.0> ** Removing (timedout) connection **

The weird stuff is that if I remove the node that's timing out (node-03 in our case), this node is able to join the cluster without any issues. If I then try to node-03 back to the cluster (the one it just left), the same issue happens on this node this time.
While this looks like a network issue, I can promise it's not, as I can see the traffic from the other VMs via tcpdump. Also, there's no firewall on the machines - they're in the same VLAN, same subnet, nothing to stop them from communicating.

Unfortunately, I'm unable to replicate it, so I guess this can be dismissed.