`command_obj` not defined.

[cggaurav]

I wonder why this happens but I am having concurrent reads at scale. "version": "2.8.0"

/root/project-x/node_modules/redis/index.js:687
    if (command_obj.error) {
                   ^

TypeError: Cannot read property 'error' of undefined
    at RedisClient.return_error (/root/project-x/node_modules/redis/index.js:687:20)
    at JavascriptRedisParser.returnError (/root/project-x/node_modules/redis/index.js:196:18)
    at JavascriptRedisParser.execute (/root/project-x/node_modules/redis-parser/lib/parser.js:572:12)
    at Socket.<anonymous> (/root/project-x/node_modules/redis/index.js:274:27)
    at emitOne (events.js:115:13)
    at Socket.emit (events.js:210:7)
    at addChunk (_stream_readable.js:266:12)
    at readableAddChunk (_stream_readable.js:253:11)
    at Socket.Readable.push (_stream_readable.js:211:10)
    at TCP.onread (net.js:585:20)

[stockholmux]

Could you provide code that is triggering this?

[BridgeAR]

@cggaurav the data used would also be very helpful. Without further information it is hard to look into this.

[jayhuang75]

@cggaurav Did you fix this issues? any suggestion? Thanks

[cggaurav]

Sorry, I can't reproduce this but seems to be randomly (not often) happening for simultaneous concurrent reads from a connection pool at scale.

[chuckwu0]

+1 , I have same error

[chuckwu0]

it happened after retry:

Stream connected groot1:8987 id 3
on_ready called groot1:8987 id 3
Net read groot1:8987 id 3
<Buffer 2d 45 52 52 20 45 4f 46 0d 0a>

[stockholmux]

@chuckwu0 What code is triggering this? That error message doesn't give us much to try and hunt it down.

[BridgeAR]

@cggaurav @chuckwu0 could you please provide me with all functions that you use and in what way including some data that you may show that you pass in?

This seems really bad if this is happening and it would be great to know where it is coming from!

[jayhuang75]

So the main reason for this happen is the Redis server down. and the node client cannot catch the error. So if you jump back the original code, put a log on it, it will show the plaintext and tell you the Redis server is down or not reachable. Hope this help!

[cggaurav]

Yes, I can confirm that this happens when the redis server is down (for instance when RDB snapshotting fails)

[BridgeAR]

Could either of you provide a small example code that would trigger this in case the redis server is down?

[patrick-oneill]

I have hit this same issue. I'm using Redis 4.0.14

This is the error seen in our application:

2019-07-01T10:38:05.09+0100 [APP/PROC/WEB/1] OUT {"name":"redis","hostname":"16098333-5ee4-4e62-4a20-eb01","pid":103,"level":30,"hostIndex":0,"dstIndex":0,"msg":"Attempting to connect to Redis via SSH tunnel","time":"2019-07-01T09:38:05.094Z","v":0}
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] OUT {"name":"redis","hostname":"16098333-5ee4-4e62-4a20-eb01","pid":103,"level":50,"err":{"command":"AUTH","code":"ERR"},"msg":"Compose Redis client error has occurred","time":"2019-07-01T09:38:05.731Z","v":0}
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] OUT {"name":"redis","hostname":"16098333-5ee4-4e62-4a20-eb01","pid":103,"level":50,"config":{"host":"127.0.0.1","port":36289,"db":12},"tunnelidx":0,"msg":"Connection to Redis server failed","time":"2019-07-01T09:38:05.731Z","v":0}
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] OUT {"name":"redis","hostname":"16098333-5ee4-4e62-4a20-eb01","pid":103,"level":50,"err":{"code":"NR_CLOSED","command":"AUTH"},"msg":"Compose Redis authentication error","time":"2019-07-01T09:38:05.734Z","v":0}
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] OUT {"name":"redis","hostname":"16098333-5ee4-4e62-4a20-eb01","pid":103,"level":50,"err":{"code":"NR_CLOSED","command":"INFO"},"msg":"Compose Redis client error has occurred","time":"2019-07-01T09:38:05.735Z","v":0}
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR /home/vcap/app/src/node_modules/redis/index.js:687
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     if (command_obj.error) {
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR                     ^
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR TypeError: Cannot read property 'error' of undefined
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at RedisClient.return_error (/home/vcap/app/src/node_modules/redis/index.js:687:21)
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at JavascriptRedisParser.returnError (/home/vcap/app/src/node_modules/redis/index.js:196:18)
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at JavascriptRedisParser.execute (/home/vcap/app/src/node_modules/redis-parser/lib/parser.js:572:12)
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at Socket.<anonymous> (/home/vcap/app/src/node_modules/redis/index.js:274:27)
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at emitOne (events.js:116:13)
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at Socket.emit (events.js:211:7)
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at addChunk (_stream_readable.js:263:12)
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at readableAddChunk (_stream_readable.js:250:11)
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at Socket.Readable.push (_stream_readable.js:208:10)
   2019-07-01T10:38:05.73+0100 [APP/PROC/WEB/1] ERR     at TCP.onread (net.js:601:20)
   2019-07-01T10:38:05.76+0100 [APP/PROC/WEB/1] ERR npm ERR! code ELIFECYCLE
   2019-07-01T10:38:05.76+0100 [APP/PROC/WEB/1] ERR npm ERR! errno 1
   2019-07-01T10:38:05.76+0100 [APP/PROC/WEB/1] ERR npm ERR! IBM-Cloud-Event-Management-Users@1.5.1 start: `cd src && node $NODE_OPTS bin/cemusers`
   2019-07-01T10:38:05.76+0100 [APP/PROC/WEB/1] ERR npm ERR! Exit status 1
   2019-07-01T10:38:05.76+0100 [APP/PROC/WEB/1] ERR npm ERR! 
   2019-07-01T10:38:05.77+0100 [APP/PROC/WEB/1] ERR npm ERR! Failed at the IBM-Cloud-Event-Management-Users@1.5.1 start script.
   2019-07-01T10:38:05.77+0100 [APP/PROC/WEB/1] ERR npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
   2019-07-01T10:38:05.77+0100 [APP/PROC/WEB/1] ERR npm WARN Local package.json exists, but node_modules missing, did you mean to install?
   2019-07-01T10:38:05.78+0100 [APP/PROC/WEB/1] ERR npm ERR! A complete log of this run can be found in:
   2019-07-01T10:38:05.78+0100 [APP/PROC/WEB/1] ERR npm ERR!     /home/vcap/app/.npm/_logs/2019-07-01T09_38_05_771Z-debug.log
   2019-07-01T10:38:05.85+0100 [APP/PROC/WEB/1] OUT Exit status 1
   2019-07-01T10:38:05.86+0100 [CELL/SSHD/1] OUT Exit status 0
   2019-07-01T10:38:08.77+0100 [CELL/0] OUT Cell 6d037dbe-fc64-4187-936c-1219fd8934c2 stopping instance f93cbfec-1769-4f23-6dda-52e8
   2019-07-01T10:38:08.77+0100 [CELL/0] OUT Cell 6d037dbe-fc64-4187-936c-1219fd8934c2 destroying container for instance f93cbfec-1769-4f23-6dda-52e8
   2019-07-01T10:38:08.98+0100 [API/13] OUT Process has crashed with type: "web"
   2019-07-01T10:38:09.00+0100 [PROXY/0] OUT Exit status 137
   2019-07-01T10:38:09.01+0100 [API/13] OUT App instance exited with guid 085b0a1d-f4d5-4fae-b158-eec68df60b72 payload: {"instance"=>"f93cbfec-1769-4f23-6dda-52e8", "index"=>0, "cell_id"=>"6d037dbe-fc64-4187-936c-1219fd8934c2", "reason"=>"CRASHED", "exit_description"=>"APP/PROC/WEB: Exited with status 1", "crash_count"=>5, "crash_timestamp"=>1561973888750471857, "version"=>"bc364e4f-d442-4638-a90a-1a7d74c40ba9"}

The scenario I was testing was a failover test for our app between two redis proxy entries.
Our app had 2 redis proxy entries configured but I configured one as a bogus entry to see if the app would failover to the second entry.

When our app starts in this configuration the above error is thrown.
The bogus entry that it is trying to connect to is an SSH tunnel to another redis deployment (which our app is not authenticated to connect to and it then throws the error.

The purpose of the test was to see how our app would behave in an outage or bad connection situation so it is a fringe case but hopefully gives you one method to recreate the issue.

I dont have sample code/apps that I can share.

[stockholmux]

@patrick-oneill Sorry, we can't help you unless you can provide the code that triggered this. The dump and failure scenario alone doesn't really tell us anything.