Reproduce incident 8197 stuck processing

[peterborbas-wise]

Context

Don't merge, just a temporal test to reproduce the issue we had with rate limits.

We had a rate limited task set to 0.5/min. The message processing lag started to increase and the rate limited task was not running. You can find more details in the incident thread.

When we removed the rate limiter, that task started to run.

I added a test to try to reproduce this issue: taskWithRateLimitingConcurrencyPolicyWillNotHangWithManyTasks
It looks like the problem is related to the number of registered tasks.
With allTasks = 10000;

• the first run was starting to process. log shows: Finished tasks: 16
• If I stop the test and start it again
• the second run stops processing: Finished tasks: 0
• every subsequent run is the same, until I drop the kafka container, then the next run works again.

Checklist

• Change meets or does not compromise the Baseline Security Requirements