Sentry consumers didn't commit offsets to Kafka consumer group and restarts regularly

[ghost]

Hello all,

I installed Sentry with Helm in EKS and used MSK to serve as external Kafka. However I am seeing sentry-generic-metrics-consumer and sentry-metrics-consumer pods restarting on a regular basis, specifically at each liveness check (every 320s). The liveness probe was okay and from logs I saw no errors. The logs simply printed partition was assigned, something like

13:58:40 [INFO] arroyo.processing.processor: New partitions assigned: {Partition(topic=Topic(name='ingest-performance-metrics'), index=0): 0} 13:58:40 [INFO] arroyo.processing.processor: Member id: 'rdkafka-9c04e400-de24-4b7b-8aaa-56d1fd8e6df5'

And from Kafka side I can see the topic has quite a bit of messages and no commits

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID generic-metrics-consumer ingest-performance-metrics 0 - 9583 - rdkafka-42179a23-6c47-4140-9b17-d419e0c26197 /10.0.38.27 rdkafka

I also tried to enable debug level log but still no errors. When container restarted the termination just showed Error code 1. Memory usage is very stable at under 500 Mi

I also tried to manually consume from the topic and I was able to consume messages

Anyone experienced similar issue? thanks !