OOM from MultithreadEventExecutorGroup under heavy load + Tasks are not correctly split between EventExecutors

[SimoneGiusso]

I have some load tests against two Spring applications that do the exact same thing but in two different ways. One is using Spring Web (blocking stack) the other is using reactive stack (Spring webflux, etc..) Both use redis: normal API in Spring Web and reactive API in Spring WebFlux.

Currently the tests are showing that the Spring Web app is capable of handling the load better than the Spring WebFlux app 🤔. It should be the opposite.

The Spring WebFlux application goes OOM under heavy load. I'm using a DefaultEventExecutorGroup with n_processors * 2 (4 in this case). This is the configuration:

    @Bean
    LettuceClientConfigurationBuilderCustomizer lettuceClientConfigurationBuilderCustomizer() {
        return builder -> builder.clientOptions(
            ClientOptions.builder()
                .disconnectedBehavior(REJECT_COMMANDS)
                .requestQueueSize(3000)
                .publishOnScheduler(true)
                .build()
        );
    }

    @Bean
    ClientResourcesBuilderCustomizer clientResourcesBuilderCustomizer() {
        return builder -> builder
            .eventExecutorGroup(new DefaultEventExecutorGroup(getRuntime().availableProcessors() * 2))
            .build();
    }

    @Bean
    @SuppressWarnings("unchecked")
    @Primary
    RedisCacheManager redisCacheManager(KryoRedisSerializer kryoRedisSerializer, LettuceConnectionFactory connectionFactory) {
        var cacheWriter = nonLockingRedisCacheWriter(connectionFactory, scan(cacheProperties.getScanCount()));
        var config = defaultCacheConfig()
            .computePrefixWith(cacheName -> format("%s:%s:", cacheProperties.getPrefix(), cacheName))
            .serializeValuesWith(SerializationPair.fromSerializer(kryoRedisSerializer));

        return fromCacheWriter(cacheWriter).cacheDefaults(config).build();
    }

    @Bean
    ReactiveRedisTemplate reactiveRedisTemplate(ReactiveRedisConnectionFactory connectionFactory, KryoRedisSerializer kryoRedisSerializer) {
        RedisSerializationContext redisSerializationContext =
            RedisSerializationContext.newSerializationContext(RedisSerializer.string())
                .value(kryoRedisSerializer).build();

        return new ReactiveRedisTemplate<String, Object>(connectionFactory, redisSerializationContext);
    }

Looking at the heap dump two things are evident:

• Around 70% of the total memory is kept by the 4 instances of DefaultEventExecutor. In particular by the taskQueue variable.
• The tasks are not split uniformly between the for instances. In fact taskQueue.count is very different in the 4 instances: 16, 17, 24'185, 64'663.

Below a screenshot of the thread dump:

For the Spring Web application I have similar configuration for the redisTemplate & redisCacheManager. In LettuceClientConfigurationBuilderCustomizer I just set the disconnectedBehaviour to REJECT_COMMANDS.

Why the Spring Web application, in particular the redis client, is working better that the WebFlux version under heavy load? I'd expect the opposite.

P.S. I specified the EventExecutorGroup to try to improve performances since by default the lettuce Reactive uses a single connection. Moreover the Netty’s threading model just one thread for each connection. In other words just one I/O event loop thread is being used for all operations (e.g. encoding, decoding, etc...). However, evidently, this didn't help.