CODEWITHSUNDEEP
redis
d-_-b
d-_-b

Reputation: 23171

redis bulk import using --pipe

I'm trying to import one million lines of redis commands, using the --pipe feature.

redis_version:2.8.1

cat file.txt | redis-cli --pipe

This results in the following error:

Error reading from the server: Connection reset by peer

Does anyone know what I'm doing wrong?

file.txt contains, for example, 

lpush name joe
lpush name bob

edit: I now see there's probably a special format(?) for using pipe mode - http://redis.io/topics/protocol

Upvotes: 12

Views: 20410

Answers (6)

deltheil
deltheil

Reputation: 16121

To use the pipe mode (a.k.a bulk loading, or mass insertion) you must indeed provide your commands directly in Redis protocol format.

The corresponding Redis protocol for LPUSH name joe is:

*3
$5
LPUSH
$4
name
$3
joe

Or as a quoted string: "*3\r\n$5\r\nLPUSH\r\n$4\r\nname\r\n$3\r\njoe\r\n".

This is what your input file must contain.

Redis documentation includes a Ruby sample to help you generate the protocol: see gen_redis_proto.

A Python sample is available e.g. in the redis-tools package.

Upvotes: 10

Rick O'Shea
Rick O'Shea

Reputation: 1449

There are existing tools that convert client commands directly to redis wire protocol messages. Example:

redis-mass my-client-script.txt | redis-cli --pipe option

https://golanglibs.com/dig_in/redis-mass https://github.com/almeida/redis-mass

Upvotes: 1

Jose Roberto Araujo
Jose Roberto Araujo

Reputation: 339

This error happens because the timeout set in Redis is Default, 0. You need to configure this timeout value by redis-cli using the command below:

To connect in redis server: redis-cli -h -p -a

To view timeout value configured: this command-line: config get timemout, Works to see what is the timeout value was configured in Redis server.

To Set new value for redis timeout: this command-line: config set timeout 120, Set the timeout to 2 minutes. So, you need to set the redis timeout so long your execution need.

I hope this answers help you. Cyu!!!

Upvotes: -1

Cédric ZUGER
Cédric ZUGER

Reputation: 422

The first point is that the parameters have to be double-quoted. The documentation is somewhat misleading on this point.

So a working syntax is : 

lpush "name" "joe"
lpush "name" "bob"

The second point is that each line has to end by an \r\n and not just by \n. To fix that point, you just have to convert your file with the command unix2dos

like : unix2dos file.txt

Then you can import your file using cat file.txt | src/redis-cli --pipe

This worked for me.

Upvotes: 15

kris.jeong
kris.jeong

Reputation: 99

There are two kinds of possibilities.

First check point is exceed of maxclients limits.

You can check using 'info clients' and 'config get maxclients' redis command.

In my desktop result is below.

127.0.0.1:6379> info clients
# Clients
connected_clients:2
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
127.0.0.1:6379> config get maxclients
1) "maxclients"
2) "2"

and then i tried to use pipe command, below is result.

[localhost redis-2.8.1]$ cat test.txt | ./src/redis-cli --pipe
All data transferred. Waiting for the last reply...
Error reading from the server: Connection reset by peer

If that result is same. you have to change redis.conf file.

Seconds check point is ulimit option.

ulimit option change needs a root privilige. check below link.

How do I change the number of open files limit in Linux?

Upvotes: 0

Santosh Joshi
Santosh Joshi

Reputation: 3320

You can use the following command to import your file's data to redis

cat file.txt | xargs  -L1 redis-cli

Upvotes: -3

Related Questions