[Questions] some producer connect to RabbitMQ timed out

[dormanze]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.6

Erlang version used

26.2.x

Operating system (distribution) used

linux

How is RabbitMQ deployed?

Generic binary package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

config and run rabbitmq

Steps to reproduce the behavior in question

producters and consumers restart

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

some producer connect to RabbitMQ timed out

2025-08-27 17:56:06.726+08:00|ERROR|gam-async-thread-5|d3861ea581f5c498|adc-agent|com.huawei.dsp.gam.inner.rabbitmq.RabbitMqProducer()|[S:CSE][T:2087][U:1699433842008109362]|Gam send msg to rabbitmq exception...
org.springframework.amqp.AmqpIOException: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<channel.open>(out-of-band=) on channel AMQChannel(amqp://[email protected]:31777//vhost_gam,34) (#34) timed out
	at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:70) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:69) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createBareChannel(CachingConnectionFactory.java:1419) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.access$200(CachingConnectionFactory.java:1405) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.doCreateBareChannel(CachingConnectionFactory.java:697) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:680) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:650) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:540) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$1600(CachingConnectionFactory.java:100) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:1424) ~[spring-rabbit-2.4.17.jar:2.4.17]
	at com.huawei.dsp.gam.inner.rabbitmq.RabbitMqProducer.send(RabbitMqProducer.java:81) ~[dsp-gam-inner-lib-1.3.2.jar:?]
	at com.huawei.dsp.gam.inner.rabbitmq.RabbitMqProducer$$FastClassBySpringCGLIB$$10bd3a86.invoke(<generated>) ~[dsp-gam-inner-lib-1.3.2.jar:?]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.39-h6.jar:5.3.39-h6]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) ~[spring-aop-5.3.39-h6.jar:5.3.39-h6]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.39-h6.jar:5.3.39-h6]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) ~[spring-aop-5.3.39-h6.jar:5.3.39-h6]
	at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:97) ~[spring-retry-1.3.4.jar:?]
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329) ~[spring-retry-1.3.4.jar:?]
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:209) ~[spring-retry-1.3.4.jar:?]
	at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:133) ~[spring-retry-1.3.4.jar:?]
	at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:159) ~[spring-retry-1.3.4.jar:?]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.39-h6.jar:5.3.39-h6]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) ~[spring-aop-5.3.39-h6.jar:5.3.39-h6]
	at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) ~[spring-aop-5.3.39-h6.jar:5.3.39-h6]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_462]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_462]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_462]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_462]
Caused by: com.rabbitmq.client.ChannelContinuationTimeoutException: Continuation call for method #method<channel.open>(out-of-band=) on channel AMQChannel(amqp://[email protected]:31777//vhost_gam,34) (#34) timed out
	at com.rabbitmq.client.impl.AMQChannel.wrapTimeoutException(AMQChannel.java:336) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:318) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:152) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:136) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:189) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:638) ~[amqp-client-5.20.0.jar:5.20.0]
	at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:58) ~[spring-rabbit-2.4.17.jar:2.4.17]
	... 26 more
Caused by: java.util.concurrent.TimeoutException: 
	at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:77) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:120) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:552) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:316) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:152) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.ChannelN.open(ChannelN.java:136) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.ChannelManager.createChannel(ChannelManager.java:189) ~[amqp-client-5.20.0.jar:5.20.0]
	at com.rabbitmq.client.impl.AMQConnection.createChannel(AMQConnection.java:638) ~[amqp-client-5.20.0.jar:5.20.0]
	at org.springframework.amqp.rabbit.connection.SimpleConnection.createChannel(SimpleConnection.java:58) ~[spring-rabbit-2.4.17.jar:2.4.17]

rabbitmq server print some notice message

Does this log indicate that my memory has exceeded the threshold?
I have set up a scheduled collection of memory usage metrics, and there have been no instances where the usage exceeded the threshold.
Additionally, there are no more detailed logs in the server logs indicating that memory usage has surpassed the threshold. I recall that in version 3.x, when memory usage exceeded the threshold, it would print the total memory currently used and the memory limit. Have we removed the printing of this information in version 4.x?

[michaelklishin]

The log messages mean that a resource (memory) alarm has gone into effect and then was cleared in a few seconds, several times. There is a doc guide for that.

The timeout in question was a channel opening timeout, which is expected on a connection that was blocked by a resource alarm (that means it has published at least one messages after the alarm has gone into effect) because RabbitMQ stops reading from the socket when a connection blocked due to a resource alarm.

[lukebakken]

Just FYI, those log messages come from memsup, and not RabbitMQ's memory monitor. This is the impetus for this PR - #14435