Importing the data ERROR

[linydquantil]

redis version: 5.0.4
redis-dump: 0.4.6

[root@mtl-gslb-redis ~]# redis-cli info
# Keyspace
db0:keys=6890,expires=0,avg_ttl=0
db1:keys=86949,expires=0,avg_ttl=0             large keys is in db1
db2:keys=1217,expires=0,avg_ttl=0
db3:keys=8196,expires=0,avg_ttl=0
db4:keys=493653,expires=0,avg_ttl=0
db5:keys=1178,expires=0,avg_ttl=0
db6:keys=1,expires=0,avg_ttl=0
db8:keys=1,expires=0,avg_ttl=0
db12:keys=155,expires=0,avg_ttl=0

redis dump success
but import data error:

[root@mtl-gslb-redis ~]# redis-cli -n 2  --pipe < 2.resp    
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 1218
[root@mtl-gslb-redis ~]# redis-cli -n 1 --pipe < 1.resp   
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
ERR Protocol error: invalid multibulk length
Error reading from the server: Connection reset by peer

[yannh]

Hi @linydquantil , I m assuming 1.resp is db1 and 2.resp is db2?

[yannh]

Could you actually give a shot at reporting against redis-cli? It looks like it might have its own timeouts... a file with garbage content that would allow to reproduce this would be super useful.
Also - could you time the failing import, and give us the size of 1.resp? How big is the single key, 200MB?

[yannh]

One simpler thing we could try is set the Zset in multiple small commands rather than a single large one. Still this is unusual :(

[linydquantil]

right, 1.resp is db1 and 2.resp is db2

[root@mtl-gslb-redis ~]# ls -lh
total 901M
-rw-r--r-- 1 root root 220K Apr 23 09:07 0.resp
-rw-r--r-- 1 root root 605M Apr 23 09:08 1.resp
-rw-r--r-- 1 root root 894K Apr 23 09:09 2.resp

The size of this key is larger than that of 200MB.

[linydquantil]

@mpartio do you have any idea?

[mpartio]

I tracked down a similar problem but I didn't find the root cause. I think it is somehow related to redis itself, not redis-dump-go. When the file-to-be-imported was sufficiently large, the import just crashed.

[yannh]

I will consider writing an improvement to import large zsets/maps in multiple commands 🙇

[yannh]

Alright I'm able to reproduce https://github.com/yannh/redis-dump-go/compare/integration-tests?expand=1
Using my mock zset I m starting to see issues once the zset has around 1M entries.

-> Filling Redis with Mock Data...
OK
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 1000000
-> Dumping DB...
Database 0: 1 element dumped
-> Flushing DB and restoring dump...
OK
Error writing to the server: Connection reset by peer``` 

[yannh]

Released v0.5.0 - long commands are now broken down into batches to avoid errors during imports.
Also: there is an acceptance tests that generates, imports, dumps a ZSET with 1M entries, and then reimports it - to ensure this does not break in the future.

[linydquantil]

this is my test result, the tool version: 0.5.0

1. get redis info

redis-cli info

# Keyspace
db0:keys=6891,expires=1,avg_ttl=48157
db1:keys=88555,expires=0,avg_ttl=0                     large keys is in db1
db2:keys=1165,expires=0,avg_ttl=0
db3:keys=8410,expires=0,avg_ttl=0
db4:keys=456667,expires=0,avg_ttl=0
db5:keys=1182,expires=0,avg_ttl=0
db6:keys=6422,expires=0,avg_ttl=0
db8:keys=1,expires=0,avg_ttl=0
db12:keys=155,expires=0,avg_ttl=0

2. dump data

[root@mtl-gslb-redis ~]# ./redis-dump-go -db 0 > 0.resp 
Database 0: 6891 element dumped
[root@mtl-gslb-redis ~]# ./redis-dump-go -db 1 > 1.resp   
Database 1: 88555 element dumped
[root@mtl-gslb-redis ~]# ./redis-dump-go -db 2 > 2.resp   
Database 2: 1165 element dumped
[root@mtl-gslb-redis ~]# ./redis-dump-go -db 3 > 3.resp   
Database 3: 8410 element dumped
[root@mtl-gslb-redis ~]# ./redis-dump-go -db 4 > 4.resp   
Database 4: 456667 element dumped
[root@mtl-gslb-redis ~]# ./redis-dump-go -db 5 > 5.resp   
Database 5: 1182 element dumped
[root@mtl-gslb-redis ~]# ./redis-dump-go -db 6 > 6.resp   
Database 6: 6422 element dumped
[root@mtl-gslb-redis ~]# ./redis-dump-go -db 8 > 8.resp   
Database 8: 1 element dumped
[root@mtl-gslb-redis ~]# ./redis-dump-go -db 12 > 12.resp   
Database 12: 155 element dumped

the resp file list is
-rw-r--r-- 1 root root 871K Apr 30 01:11 0.resp
-rw-r--r-- 1 root root  56K Apr 30 01:18 12.resp
-rw-r--r-- 1 root root 491M Apr 30 01:12 1.resp
-rw-r--r-- 1 root root 754K Apr 30 01:14 2.resp
-rw-r--r-- 1 root root  14M Apr 30 01:14 3.resp
-rw-r--r-- 1 root root  33M Apr 30 01:17 4.resp
-rw-r--r-- 1 root root 1.1M Apr 30 01:18 5.resp
-rw-r--r-- 1 root root 645K Apr 30 01:18 6.resp
-rw-r--r-- 1 root root  796 Apr 30 01:18 8.resp

3. flushall redis

redis-cli flushall

4. import data

[root@mtl-gslb-redis ~]# redis-cli  -n 0 --pipe < 0.resp 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 6893
[root@mtl-gslb-redis ~]# redis-cli -n 1 --pipe < 1.resp 
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 100859
[root@mtl-gslb-redis ~]# redis-cli  -n 2 --pipe < 2.resp   
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 1166
[root@mtl-gslb-redis ~]# redis-cli -n 3 --pipe < 3.resp   
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 8411
[root@mtl-gslb-redis ~]# redis-cli  -n 4 --pipe < 4.resp   
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 456668
[root@mtl-gslb-redis ~]# redis-cli  -n 5 --pipe < 5.resp    
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 1183
[root@mtl-gslb-redis ~]# redis-cli  -n 6 --pipe < 6.resp   
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 6423
[root@mtl-gslb-redis ~]# redis-cli -n 8 --pipe < 8.resp   
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 2
[root@mtl-gslb-redis ~]# redis-cli -n 12 --pipe < 12.resp    
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 156

5. get redis info

# Keyspace
db0:keys=6890,expires=0,avg_ttl=0
db1:keys=88555,expires=0,avg_ttl=0                     The number of key is the same.
db2:keys=1165,expires=0,avg_ttl=0
db3:keys=8410,expires=0,avg_ttl=0
db4:keys=456667,expires=0,avg_ttl=0
db5:keys=1182,expires=0,avg_ttl=0
db6:keys=6422,expires=0,avg_ttl=0
db8:keys=1,expires=0,avg_ttl=0
db12:keys=155,expires=0,avg_ttl=0

[linydquantil]

@yannh @mpartio
i use v0.5.0 to do some tests. this tool is good for me
Next, I will do a more detailed test.
thank you for your help

[linydquantil]

how to get redis-dump-go version? Maybe it needs to support functions like this: ./redis-dump-go --version.

[yannh]

Yes, that would be nice, thanks for the suggestion :) Might work on this at some point, would also accept a patch!

[linydquantil]

@yannh i found the same error when the redis version is 6.2.5

redis version: 6.2.5
redis dump go: v0.5.1

[cdb_dev_1109]# redis-cli -n 1 --pipe <  cdb_db_1.resp
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
ERR Protocol error: invalid multibulk length

can you help me check it ?