3.13.7 Shovel's rabbit_shovel_dyn_worker_sup_sup can fail without being restarted (exceed supervisor restart intensity?)

[kubrakaraman6]

Describe the bug

Environment

• RabbitMQ Version: 3.13.7
• **Erlang (SMP,ASYNC_THREADS) (BEAM) emulator version 14.2.5.10
• Cluster Type: Multi-node ( 3 node )
• OS: (e.g., Ubuntu 22.04 )
• Plugins Enabled:
  • rabbitmq_shovel
  • rabbitmq_shovel_management

We observed that a dynamic shovel supervisor process (rabbit_shovel_dyn_worker_sup_sup) on one cluster node crashes and does not recover automatically.
The affected node (rabbit@rabbit-01) cannot start any shovel workers until the node or shovel application is manually restarted.
Meanwhile, the same shovel definition works successfully when created on the second node (rabbit@rabbit-02).
[error] <0.211588995.1> Shovel with the name 'Move from Queque_error' was not found on virtual host '/'
[error] <0.211588995.1> Could not find shovel data for shovel 'Move from Queque_error' in vhost: '/'
[error] <0.211588995.1> Failed to delete shovel 'Move from Queque_error' on vhost '/', reason: {exception,
{noproc,
{gen_server,
call,
[rabbit_shovel_dyn_worker_sup_sup,which_children,infinity]}}}
[info] <0.211582083.1> Waiting for Mnesia tables for 30000 ms, 9 retries left

2025-10-23 05:31:49.714145+00:00 [info] <0.211347036.1> Waiting for Mnesia tables for 30000 ms, 9 retries left
2025-10-23 05:31:49.714264+00:00 [info] <0.211347036.1> Successfully synced tables from a peer
2025-10-23 05:31:57.144238+00:00 [warning] <0.167982866.1> Recurring shovel spec clean up failed with exit:{noproc,
2025-10-23 05:31:57.144238+00:00 [warning] <0.167982866.1> {gen_server,call,
2025-10-23 05:31:57.144238+00:00 [warning] <0.167982866.1> [rabbit_shovel_dyn_worker_sup_sup,
2025-10-23 05:31:57.144238+00:00 [warning] <0.167982866.1> which_children,infinity]}}
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> crasher:
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> initial call: cowboy_stream_h:request_process/3
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> pid: <0.211166023.1>
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> registered_name: []
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> exception exit: {{noproc,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {gen_server,call,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> [rabbit_shovel_dyn_worker_sup_sup,which_children,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> infinity]}},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> [{gen_server,call,3,[{file,"gen_server.erl"},{line,419}]},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {mirrored_supervisor,child,2,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> [{file,"mirrored_supervisor.erl"},{line,226}]},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {mirrored_supervisor,call,2,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> [{file,"mirrored_supervisor.erl"},{line,203}]},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {mirrored_supervisor,fold,3,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> [{file,"mirrored_supervisor.erl"},{line,219}]},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {rabbit_shovel_dyn_worker_sup_sup,child_exists,1,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> [{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {line,69}]},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {rabbit_shovel_dyn_worker_sup_sup,adjust,2,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> [{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {line,34}]},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {rabbit_runtime_parameters,set_any0,5,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> [{file,"rabbit_runtime_parameters.erl"},{line,147}]},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {rabbit_runtime_parameters,set_any,5,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> [{file,"rabbit_runtime_parameters.erl"},
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> {line,122}]}]}
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> in function gen_server:call/3 (gen_server.erl, line 419)
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> in call from mirrored_supervisor:child/2 (mirrored_supervisor.erl, line 226)
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> in call from mirrored_supervisor:call/2 (mirrored_supervisor.erl, line 203)
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> in call from mirrored_supervisor:fold/3 (mirrored_supervisor.erl, line 219)
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> in call from rabbit_shovel_dyn_worker_sup_sup:child_exists/1 (rabbit_shovel_dyn_worker_sup_sup.erl, line 69)
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> in call from rabbit_shovel_dyn_worker_sup_sup:adjust/2 (rabbit_shovel_dyn_worker_sup_sup.erl, line 34)
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> in call from rabbit_runtime_parameters:set_any0/5 (rabbit_runtime_parameters.erl, line 147)
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> in call from rabbit_runtime_parameters:set_any/5 (rabbit_runtime_parameters.erl, line 122)
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> ancestors: [<0.211394560.1>,<0.168020937.1>,<0.167981760.1>,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> <0.167984701.1>,<0.168020606.1>,rabbit_web_dispatch_sup,
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> <0.628.0>]
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> message_queue_len: 0
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> messages: []
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> links: [<0.211394560.1>]
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> dictionary: []
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> trap_exit: false
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> status: running
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> heap_size: 2586
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> stack_size: 28
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> reductions: 4435
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1> neighbours:
2025-10-23 05:32:43.820388+00:00 [error] <0.211166023.1>
2025-10-23 05:32:43.820961+00:00 [error] <0.211394560.1> Ranch listener {acceptor,{0,0,0,0,0,0,0,0},15672}, connection process <0.211394560.1>, stream 1 had its request process <0.211166023.1> exit with reason {noproc,{gen_server,call,[rabbit_shovel_dyn_worker_sup_sup,which_children,infinity]}} and stacktrace [{gen_server,call,3,[{file,"gen_server.erl"},{line,419}]},{mirrored_supervisor,child,2,[{file,"mirrored_supervisor.erl"},{line,226}]},{mirrored_supervisor,call,2,[{file,"mirrored_supervisor.erl"},{line,203}]},{mirrored_supervisor,fold,3,[{file,"mirrored_supervisor.erl"},{line,219}]},{rabbit_shovel_dyn_worker_sup_sup,child_exists,1,[{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},{line,69}]},{rabbit_shovel_dyn_worker_sup_sup,adjust,2,[{file,"rabbit_shovel_dyn_worker_sup_sup.erl"},{line,34}]},{rabbit_runtime_parameters,set_any0,5,[{file,"rabbit_runtime_parameters.erl"},{line,147}]},{rabbit_runtime_parameters,set_any,5,[{file,"rabbit_runtime_parameters.erl"},{line,122}]}]
2025-10-23 05:32:43.820961+00:00 [error] <0.211394560.1>
2025-10-23 05:32:57.145259+00:00 [warning] <0.167982866.1> Recurring shovel spec clean up failed with exit:{noproc,
2025-10-23 05:32:57.145259+00:00 [warning] <0.167982866.1> {gen_server,call,
2025-10-23 05:32:57.145259+00:00 [warning] <0.167982866.1> [rabbit_shovel_dyn_worker_sup_sup,
2025-10-23 05:32:57.145259+00:00 [warning] <0.167982866.1> which_children,infinity]}}
2025-10-23 05:33:57.146258+00:00 [warning] <0.167982866.1> Recurring shovel spec clean up failed with exit:{noproc,
2025-10-23 05:33:57.146258+00:00 [warning] <0.167982866.1> {gen_server,call,
2025-10-23 05:33:57.146258+00:00 [warning] <0.167982866.1> [rabbit_shovel_dyn_worker_sup_sup,
2025-10-23 05:33:57.146258+00:00 [warning] <0.167982866.1> which_children,infinity]}}
2025-10-23 05:34:57.147260+00:00 [warning] <0.167982866.1> Recurring shovel spec clean up failed with exit:{noproc,
2025-10-23 05:34:57.147260+00:00 [warning] <0.167982866.1> {gen_server,call,
2025-10-23 05:34:57.147260+00:00 [warning] <0.167982866.1> [rabbit_shovel_dyn_worker_sup_sup,
2025-10-23 05:34:57.147260+00:00 [warning] <0.167982866.1> which_children,infinity]}}
2025-10-23 05:35:14.815840+00:00 [info] <0.211412137.1> Waiting for Mnesia tables for 30000 ms, 9 retries left
2025-10-23 05:35:14.815968+00:00 [info] <0.211412137.1> Successfully synced tables from a peer
2025-10-23 05:35:22.407824+00:00 [info] <0.211412483.1> Waiting for Mnesia tables for 30000 ms, 9 retries left
2025-10-23 05:35:22.407949+00:00 [info] <0.211412483.1> Successfully synced tables from a peer
2025-10-23 05:35:23.590839+00:00 [info] <0.211412854.1> Waiting for Mnesia tables for 30000 ms, 9 retries left

After this point:

rabbitmqctl eval 'supervisor2:which_children(rabbit_shovel_dyn_worker_sup_sup).'
# => {:noproc, {:gen_server, :call, [:rabbit_shovel_dyn_worker_sup_sup, :which_children, :infinity]}}



### Reproduction steps

Create a dynamic shovel in vhost / (e.g. “Move from Queque_error”).

Trigger a cluster sync event (e.g., restart one node or cause Mnesia table resync).

Attempt to delete and recreate the shovel on the affected node (rabbit@rabbit-01).

Observe noproc errors in logs and that the supervisor process no longer exists.

Apply the same shovel on another node (rabbit@rabbit-02) → shovel works correctly.

### Expected behavior

The shovel supervisor should remain running or automatically recover after Mnesia sync or parameter update.
New shovels should start normally on all nodes without requiring a restart.

### Additional context

On one node, the shovel supervisor crashes and stays dead.

New shovel definitions cannot start workers.

The same definition works fine on a different cluster node.

Only restarting the rabbitmq_shovel application or node recovers functionality.

[michaelklishin]

@kubrakaraman6 RabbitMQ 3.13.x is out of community support.

But the Same Fundamental Problem Applies to 4.x?

That's correct, 4.x still uses mirrored_supervisor and it hosts all shovels on a single node.
Like all Mnesia-based features, mirrored_supervisor's fundamental limitations are considered to be unfixable by our team. You can only replace it with something else, like Mnesia was replaced by/with Khepri and new 4.2.0 clusters will use Khepri by default.

This is one of the reasons why in Tanzu RabbitMQ 4.2.0, we have a new plugin that offers distributed shovels that, as the name suggests, distributes shovels on all nodes. As you can imagine, the key corporate sponsor of RabbitMQ is not going to open source that work any time soon, or likely ever.

What that means for rabbitmq_shovel we have in the open source edition, time will tell. Its current limitations are perfectly acceptable for a lot of users, as rabbitmq_shovel's 16 year long history proves.

Those who want distributed shovels will have to get a Tanzu RabbitMQ license. This is the collective price we pay for core RabbitMQ being open source and a very significant amount of effort being invested into the open source edition.

[michaelklishin]

Only restarting the rabbitmq_shovel application or node recovers functionality.

This means that in your specific case, it likely was only the local rabbit_shovel_dyn_worker_sup_sup process that has terminated. There are scenarios that are worse than that, where you'd have to restart a specific node, but they are extremely rare to see.

This could potentially be a matter of configuring the supervisor tree restart intensity settings around rabbit_shovel_dyn_worker_sup_sup, for example, its parent supervisor is rabbit_shovel_sup.

And it just happens to be the case that Tanzu RabbitMQ 4.2.0 distributed shovels address this specific scenario as well, even though it wasn't the reason why distributed shovels were originally developed, and the goal was not to make regular shovels worse (in fact, 4.2.0 ships with regular shovels improvements such as the local "protocol" support).

Be that as it may, a significant shovel or mirrored_supervisor design changes are not in the cards for open source RabbitMQ, whether we like it or not. Everything else around shovels in the open source edition is still developed and maintained as any other feature.