CODEWITHSUNDEEP
pythonredis
Peter
Peter

Reputation: 1253

Вest practise to send big amount of data to Redis

I use python3.6 and Redis version=5.0.7. And I'm trying to send about 40 Mb to Redis. I have following error ConnectionResetError: [Errno 104] Connection reset by peer

How to overcome this issue and what the best practices to send lots of data to Redis? I would be grateful for any advice.

import hashlib
import sys
import redis
client = redis.StrictRedis.from_url('redis://redis:6379/10', encoding="utf-8", decode_responses=True)

gen_data = {}
for i in range(10 ** 6):
    import random
    sku = f'{str(random.random())[:8]}-{random.random()}'
    hash = hashlib.md5(sku.encode()).hexdigest()
    gen_data[sku] = hash

print(f"len: {len(gen_data)} size: {sys.getsizeof(gen_data)/(1024*1024)} Mb")
client.hmset('products', gen_data)

Output

len: 1000000 size: 40.000099182128906 Mb
Traceback (most recent call last):
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/connection.py", line 666, in send_packed_command
    sendall(self._sock, item)
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/_compat.py", line 8, in sendall
    return sock.sendall(*args, **kwargs)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/peter/projects/vostok_projects/catalog/temp.py", line 16, in <module>
    client.hmset('products', gen_data)
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/client.py", line 2761, in hmset
    return self.execute_command('HMSET', name, *items)
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/client.py", line 838, in execute_command
    conn.send_command(*args)
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/connection.py", line 687, in send_command
    check_health=kwargs.get('check_health', True))
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/connection.py", line 679, in send_packed_command
    (errno, errmsg))
redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.

Upvotes: 2

Views: 1179

Answers (1)

Peter
Peter

Reputation: 1253

My thoughts about question:

Don't allow such a large amount of data at all. Redis is a single-threaded, hence such request will suspend Redis. If it really needs to do it you should use batch request.

import itertools

def dict_to_chunks(data, size=10000):
    it = iter(data)
    for _ in range(0, len(data), size):
        yield {k: data[k] for k in itertools.islice(it, size)}


for d in dict_to_chunks(gen_data):
    client.hmset('products', d)

Also there is an ability to store up to 512 Mb serialized binary string as Redis' key. Read and write will be relatively fast.

Upvotes: 1

Related Questions