CODEWITHSUNDEEP
pythonsocketsredisredis-py
chandu
chandu

Reputation: 413

Broken Pipe Error Redis

We are trying to SET pickled object of size 2.3GB into redis through redis-py package. Encountered the following error.

BrokenPipeError: [Errno 32] Broken pipe

redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.

I would like to understand the root cause. Is it due to input/output buffer limitation at server side or client side ? Is it due to any limitations on RESP protocol? Is single value (bytes) of 2.3 Gb allowed to store into Redis ?

import redis

r = redis.StrictRedis(host='10.X.X.X', port=7000, db=0)

pickled_object = pickle.dumps(obj_to_be_pickled)

r.set('some_key', pickled_object)

Client Side Error

BrokenPipeError: [Errno 32] Broken pipe

/usr/local/lib/python3.4/site-packages/redis/connection.py(544)send_packed_command()

self._sock.sendall(item)

Server Side Error

31164:M 04 Apr 06:02:42.334 - Protocol error from client: id=95 addr=10.2.130.144:36120 fd=11 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=16384 qbuf-free=16384 obl=42 oll=0 omem=0 events=r cmd=NULL

31164:M 04 Apr 06:07:09.591 - Protocol error from client: id=96 addr=10.2.130.144:36139 fd=11 name= age=9 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=40 qbuf-free=32728 obl=42 oll=0 omem=0 events=r cmd=NULL

Redis Version : 3.2.8 / 64 bit

Upvotes: 7

Views: 16133

Answers (2)

chandu
chandu

Reputation: 413

The issue is with the data size being passed to Redis. The command is sent to Redis as two items follows the RESP Standards

item #1 

b'*3\r\n$3\r\nSET\r\n$8\r\nsome_key\r\n$2460086692\r\n'

Where
    *3           - indicates RESP array of three elements
    \r\n         - indicates the RESP Carriage return Line Feeder(separator)
    $3           - indicates Bulk string of length 3 bytes(here it is 'SET')
    $8           - indicates Bulk String of length 8 bytes(he it is 'some_key')
    $2460086692  - indicates Bulk String of length 2460086692 bytes (the length of value 2460 MB to be passed to Redis as next item )

item #2 

b'\x80\x03csklearn.ensemble.forest\nRandomForestC...

Here item #2 indicates the actual data
  • The moment item #1 instruction is passed to Redis Server, the server closed the connection as the value $2460086692 violated the protocol rule of 512 MB
  • When the item #2 is sent to the Redis Server, we got Broken Pipe exception as the connection is already closed by the server.

Upvotes: 6

Itamar Haber
Itamar Haber

Reputation: 49932

Redis' String data type can be at most 512MB.

Upvotes: 11

Related Questions