CODEWITHSUNDEEP
pythonredismultiprocessing
user998476
user998476

Reputation: 177

Redis pipelines and python multiprocessing

Would it be possible to execute multiple redis pipelines in parallel using the python multiprocessing module to increase redis throughput?

Upvotes: 2

Views: 3281

Answers (3)

The Real Bill
The Real Bill

Reputation: 15823

In order to use it with python's multiprocessing model you will need to create a new connection in each subprocess to ensure each process has it's own connection. Otherwise you can run into contention issues on the client side.

That said, if there are commands you need to run as a transaction you will want to use multi/exec yourself as pipelining is not the same thing and does not call it. The simplest way with py-redis is by setting the transaction flag to True when calling pipeline. But only do this if you really need every other client to wait for that pipeline to finish executing. If you do that you've essentially made your application non-threaded as it works like a lock on the database - all other clients can't operate on the database while a MULTI/EXEC is in play.

If you must use MULTI/EXEC and still want the concurrency you will need to isolate groups of keys on different servers and run a server per connection needing to lock the DB. If your operations are on keys which have overlap in various processes, this will require either accepting the effects of MULTI/EXEC on the overall performance or redesigning the client code to eliminate the contention.

Upvotes: 2

irmorteza
irmorteza

Reputation: 1654

Of course it's possible. Pipeline is a simple transaction that works like executing batch queries at the same time. It uses MULTI / EXEC in code behind. By pipeline, Redis will queue commands. And all the commands are executed once EXEC is called. (pipeline.flush()).

This it just for ensuring of executing some commands while some unwanted event may failure(closing client connection)

I used pipeline in multi threads, and there was no problem. But I recommended if you are processing on some heavy stuffs simultaneously ,It would be better to use multi instances (if it's possible to separate them) or different connections too.

Upvotes: 1

Itamar Haber
Itamar Haber

Reputation: 50102

An answer has to consist of at least 30 characters so mine is: "yes".

Upvotes: 3

Related Questions