[Questions] rabbitmqctl run failed with a net kernel exception

[dormanze]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

other (3.13.0)

Erlang version used

26.2.x

Operating system (distribution) used

SUSE Linux

How is RabbitMQ deployed?

Generic binary package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics
rabbitmqctl print many error:

09:36:49.775 [error] rabbit_env: Failed to setup distribution (as rabbit_ctl_29@test-rabbitmqservice-0) to query node rabbit@test-rabbitmqservice-0: {:error,
 {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}},
  {:child, :undefined, :net_sup_dynamic,
   {:erl_distribution, :start_link,
    [
      %{
        name: :"rabbit_ctl_29@test-rabbitmqservice-0",
        supervisor: :net_sup_dynamic,
        net_tickintensity: 4,
        net_ticktime: 60,
        name_domain: :shortnames,
        clean_halt: false
      }
    ]}, :permanent, false, 1000, :supervisor, [:erl_distribution]}}}

09:36:49.800 [error] rabbit_env: Failed to setup distribution (as rabbit_ctl_63@test-rabbitmqservice-0) to query node rabbit@test-rabbitmqservice-0: {:error,
 {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}},
  {:child, :undefined, :net_sup_dynamic,
   {:erl_distribution, :start_link,
    [
      %{
        name: :"rabbit_ctl_63@test-rabbitmqservice-0",
        supervisor: :net_sup_dynamic,
        net_tickintensity: 4,
        net_ticktime: 60,
        name_domain: :shortnames,
        clean_halt: false
      }
    ]}, :permanent, false, 1000, :supervisor, [:erl_distribution]}}}

09:36:49.801 [error] rabbit_env: Failed to setup distribution (as rabbit_ctl_62@test-rabbitmqservice-0) to query node rabbit@test-rabbitmqservice-0: {:error,
 {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}},
  {:child, :undefined, :net_sup_dynamic,
   {:erl_distribution, :start_link,
    [
      %{
        name: :"rabbit_ctl_62@test-rabbitmqservice-0",
        supervisor: :net_sup_dynamic,
        net_tickintensity: 4,
        net_ticktime: 60,
        name_domain: :shortnames,
        clean_halt: false
      }
    ]}, :permanent, false, 1000, :supervisor, [:erl_distribution]}}}
................

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs
rabbitmq server start failed

2025-07-03 10:46:14.076211+08:00 [notice] <0.136.0> maybe_setup_dist_for_remote_query 3
2025-07-03 10:46:14.609285+08:00 [notice] <0.153.0> Protocol 'inet_tcp': register/listen error: epmd_close
2025-07-03 10:46:14.609285+08:00 [notice] <0.153.0> 
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>   crasher:
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     initial call: net_kernel:init/1
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     pid: <0.153.0>
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     registered_name: []
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     exception exit: {error,badarg}
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>       in function  gen_server:init_it/6 (gen_server.erl, line 961)
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     ancestors: [net_sup,kernel_sup,<0.47.0>]
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     message_queue_len: 0
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     messages: []
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     links: [<0.150.0>,#Port<0.10>]
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     dictionary: [{longnames,false}]
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     trap_exit: true
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     status: running
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     heap_size: 1598
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     stack_size: 28
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>     reductions: 2793
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0>   neighbours:
2025-07-03 10:46:14.609388+08:00 [error] <0.153.0> 
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0>     supervisor: {local,net_sup}
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0>     errorContext: start_error
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0>     reason: {'EXIT',nodistribution}
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0>     offender: [{pid,undefined},
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0>                {id,net_kernel},
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0>                {restart_type,permanent},
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0>                {significant,false},
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0>                {shutdown,2000},
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0>                {child_type,worker}]
2025-07-03 10:46:14.612547+08:00 [error] <0.150.0> 

BOOT FAILED
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0> 
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0> BOOT FAILED
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0> ===========
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0> Exception during startup:
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0> 
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0> error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{name => rabbit_prelaunch_61831@localhost,supervisor => net_sup_dynamic,net_tickintensity => 4,net_ticktime => 60,name_domain => shortnames,clean_halt => false}]},permanent,false,1000,supervisor,[erl_distribution]}}}}
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0> 
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0>     rabbit_prelaunch_dist:duplicate_node_check/1, line 80
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0>     rabbit_prelaunch_dist:setup/1, line 23
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0>     rabbit_prelaunch:do_run/0, line 115
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0>     rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0>     supervisor:do_start_child_i/3, line 420
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0>     supervisor:do_start_child/2, line 406
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0>     supervisor:-start_children/2-fun-0-/3, line 390
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0>     supervisor:children_map/4, line 1258
2025-07-03 10:46:14.614168+08:00 [error] <0.136.0> 
===========
Exception during startup:

error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{name => rabbit_prelaunch_61831@localhost,supervisor => net_sup_dynamic,net_tickintensity => 4,net_ticktime => 60,name_domain => shortnames,clean_halt => false}]},permanent,false,1000,supervisor,[erl_distribution]}}}}

    rabbit_prelaunch_dist:duplicate_node_check/1, line 80
    rabbit_prelaunch_dist:setup/1, line 23
    rabbit_prelaunch:do_run/0, line 115
    rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
    supervisor:do_start_child_i/3, line 420
    supervisor:do_start_child/2, line 406
    supervisor:-start_children/2-fun-0-/3, line 390
    supervisor:children_map/4, line 1258

2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>     supervisor: {local,rabbit_prelaunch_sup}
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>     errorContext: start_error
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>     reason: {badmatch,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                 {error,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                     {{shutdown,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                          {failed_to_start_child,net_kernel,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                              {'EXIT',nodistribution}}},
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                      {child,undefined,net_sup_dynamic,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                          {erl_distribution,start_link,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                              [#{name => rabbit_prelaunch_61831@localhost,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                                 supervisor => net_sup_dynamic,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                                 net_tickintensity => 4,net_ticktime => 60,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                                 name_domain => shortnames,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                                 clean_halt => false}]},
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                          permanent,false,1000,supervisor,
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                          [erl_distribution]}}}}
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>     offender: [{pid,undefined},
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                {id,prelaunch},
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                {restart_type,transient},
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                {significant,false},
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                {shutdown,5000},
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0>                {child_type,worker}]
2025-07-03 10:46:15.615173+08:00 [error] <0.136.0> 
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>   crasher:
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     initial call: application_master:init/4
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     pid: <0.134.0>
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     registered_name: []
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     exception exit: {{shutdown,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                       {failed_to_start_child,prelaunch,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                        {badmatch,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                         {error,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                          {{shutdown,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                            {failed_to_start_child,net_kernel,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                             {'EXIT',nodistribution}}},
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                           {child,undefined,net_sup_dynamic,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                            {erl_distribution,start_link,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                             [#{name => rabbit_prelaunch_61831@localhost,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                                supervisor => net_sup_dynamic,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                                net_tickintensity => 4,net_ticktime => 60,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                                name_domain => shortnames,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                                clean_halt => false}]},
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                            permanent,false,1000,supervisor,
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                            [erl_distribution]}}}}}},
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>                      {rabbit_prelaunch_app,start,[normal,[]]}}
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>       in function  application_master:init/4 (application_master.erl, line 142)
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     ancestors: [<0.133.0>]
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     message_queue_len: 1
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     messages: [{'EXIT',<0.135.0>,normal}]
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     links: [<0.133.0>,<0.44.0>]
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     dictionary: []
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     trap_exit: true
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     status: running
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     heap_size: 376
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     stack_size: 28
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>     reductions: 165
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0>   neighbours:
2025-07-03 10:46:15.615650+08:00 [error] <0.134.0> 
2025-07-03 10:46:15.616154+08:00 [notice] <0.44.0> Application rabbitmq_prelaunch exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{name => rabbit_prelaunch_61831@localhost,supervisor => net_sup_dynamic,net_tickintensity => 4,net_ticktime => 60,name_domain => shortnames,clean_halt => false}]},permanent,false,1000,supervisor,[erl_distribution]}}}}}},{rabbit_prelaunch_app,start,[normal,[]]}}
Runtime terminating during boot (terminating)

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

listeners.tcp = none
listeners.ssl.default = 172.26.25.2:5673
credential_validator.validation_backend = rabbit_credential_validator_password_regexp
credential_validator.regexp = ^(?=.*\d)(?=.*[A-Z])(?=.*[a-z])(?=.*[~@%^_=+\[{}\]:,.\/?])[\da-zA-Z~@%^_=+\[{}\]:,.\/?]{16,32}$|guest
log.file.level = info
log.file.rotation.size = 524288000
log.file.rotation.count = 0
vm_memory_high_watermark.absolute = 1638MiB
password_hashing_module = rabbit_password_hashing_sha512
cluster_partition_handling = autoheal
cluster_formation.discovery_retry_limit = 20
cluster_formation.discovery_retry_interval = 1000
mnesia_table_loading_retry_timeout = 30000
mnesia_table_loading_retry_limit = 5

Steps to deploy RabbitMQ cluster

setp1. set rabbitmq node name to /etc/hosts:

# Kubernetes-managed hosts file (host network).
#
# hosts         This file describes a number of hostname-to-address
#               mappings for the TCP/IP subsystem.  It is mostly
#               used at boot time, when no name servers are running.
#               On small systems, this file can be used instead of a
#               "named" name server.
# Syntax:
#    
# IP-Address  Full-Qualified-Hostname  Short-Hostname
#

127.0.0.1       localhost

# special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts
127.0.0.1 hadoop.hadoop.com
127.0.0.1 paas-core
127.0.0.1 paas-172-26-25-2
192.168.25.3 registry.fusionstage.local
172.26.25.2 test-rabbitmqservice-0

setp2. init rabbitmq config

listeners.tcp = none
listeners.ssl.default = 172.26.25.2:5673
credential_validator.validation_backend = rabbit_credential_validator_password_regexp
credential_validator.regexp = ^(?=.*\d)(?=.*[A-Z])(?=.*[a-z])(?=.*[~@%^_=+\[{}\]:,.\/?])[\da-zA-Z~@%^_=+\[{}\]:,.\/?]{16,32}$|guest
log.file.level = info
log.file.rotation.size = 524288000
log.file.rotation.count = 0
vm_memory_high_watermark.absolute = 1638MiB
password_hashing_module = rabbit_password_hashing_sha512
cluster_partition_handling = autoheal
cluster_formation.discovery_retry_limit = 20
cluster_formation.discovery_retry_interval = 1000
mnesia_table_loading_retry_timeout = 30000
mnesia_table_loading_retry_limit = 5

step3. run rabbitmq-server

Steps to reproduce the behavior in question

just run rabbitmqctl

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

Based on the error message, I located the call location in the code(in rabbit_env.erl):

    case net_kernel:start([Nodename, NameType]) of
        {ok, _} ->
            update_context(Context, dist_started_for_remote_query, true);
        {error, {already_started, _}} ->
            Context;
        {error, {{already_started, _}, _}} ->
            Context;
        Error ->
            logger:error(
              "rabbit_env: Failed to setup distribution (as ~ts) to "
              "query node ~ts: ~tp",
              [Nodename, Remote, Error]),
            setup_dist_for_remote_query(Context,
                                        NamePart, HostPart, NameType,
                                        Attempts - 1)
    end.

I checked my node name and the information in /etc/hosts based on information from other forums and found nothing unusual. Please provide some ideas for solving the problem. At the same time, after I changed the calling node, it started normally.
The /etc/hosts file of a normally invoked node is as follows:

# Kubernetes-managed hosts file (host network).
#
# hosts         This file describes a number of hostname-to-address
#               mappings for the TCP/IP subsystem.  It is mostly
#               used at boot time, when no name servers are running.
#               On small systems, this file can be used instead of a
#               "named" name server.
# Syntax:
#    
# IP-Address  Full-Qualified-Hostname  Short-Hostname
#

127.0.0.1       localhost

# special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts


127.0.0.1 kube-system-fi-node1
192.168.25.3 registry.fusionstage.local
203.0.113.0 hadoop.hadoop.com hadoop.hadoop.com.
203.0.113.0 manager manager.
172.26.25.9 fi-node2 fi-node2.
172.26.25.10 fi-node3 fi-node3.
172.26.25.7 fi-node1 fi-node1.


192.168.25.3 csms-storagemgr.fst-manage.svc.cluster.local
172.26.25.7 test-rabbitmqservice-0

[michaelklishin]

RabbitMQ 3.13.x has been out of community support for some 10 months now.

Something in your environment violates the fundamental assumptions for the protocol nodes and CLI tools use.