Laravel Horizon generating CROSSSLOT error with AWS Redis

[aaflatooni]

We are getting CROSSSLOT errors from time to time using AWS Redis.
{
   "class": "Predis\Response\ServerException",
   "message": "CROSSSLOT Keys in request don't hash to the same slot",
   "code": 0,
   "file": "/var/app/current/vendor/predis/predis/src/Client.php:370",
   "trace": [
       "/var/app/current/vendor/predis/predis/src/Client.php:335",
       "/var/app/current/vendor/predis/predis/src/Client.php:314",
       "/var/app/current/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php:116",
       "/var/app/current/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php:220",
       "/var/app/current/vendor/laravel/horizon/src/Repositories/RedisMetricsRepository.php:200",

AWS Redis is just 1 node and 1 shard.

We have redis defined in configu/queue.php:
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
]

Redis in config/database.php:
'redis' => [

    'client' => env('REDIS_CLIENT', 'phpredis'),

    'options' => [
        'cluster' => env('REDIS_CLUSTER', 'redis'),
        'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
    ],
    'default' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_DB', '0'),
    ],
    'cache' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', '6379'),
        'database' => env('REDIS_CACHE_DB', '1'),
    ],
],

Any suggestions on how we can avoid the CROSSSLOT error when using the AWS Redis?

[raphaelcangucu]

Anyone ?

[aaflatooni]

We managed to resolve it.

We had to put the queue names in the horizon.conf in {}

So as an example:
'queue' => ['{default}','{queue2}','{queue3}'],

You should make sure that all of them have {} in horizon.conf so as an example:

'prefix' => '{aaredis}:',

[raphaelcangucu]

Hi @aaflatooni ,

Thanks a lot, it worked.

Here is my database.php file:

'redis' => [

        'client' => env('REDIS_CLIENT', 'predis'),

        'options' => [
            'cluster' => env('REDIS_CLUSTER', 'redis'),
            'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
        ],

And I just needed to change my .ENV to add the prefix as REDIS_PREFIX="{gamba}"

[b8ne]

I've just tried this. It removes the crosslot error but jobs are defaulting to 'default' queue, and get stuck in pending. Any ideas?

[nikolaynesov]

While it’s technically possible to globally prefix all keys with a shared hash slot to avoid CROSSSLOT errors, doing so would undermine the core benefits of clustering, including horizontal scaling and balanced distribution across nodes. We need another solution. Reviewing every Redis communication in the app and using slots properly.

[aaflatooni]

Not all the jobs should end up in Redis. Redis is expensive and jobs could get trimmed silently if it is busy. The jobs might not need the performance that Redis provides. It would be great if Horizon could work with another DB like MySQL.