CODEWITHSUNDEEP
pythonperformancerediszeromqgevent
pedrotech
pedrotech

Reputation: 1379

ZeroMQ, Redis and Gevent

I'm developing a server which has to process a big amount of small data requests.
The server is developed with ZeroMQ, using the PULL/PUSH pattern.
Basically each request process consists in storing its data (message) in Redis.

I would like to have better performance by processing each request with gevent.
The following code doesn't work (messages are not stored in Redis).

from gevent import monkey
monkey.patch_all()
from redis import Redis, StrictRedis
from redis import connection
import zmq
import gevent


context = zmq.Context()
socket = context.socket(zmq.PULL)
socket.bind("tcp://*:5000")

connection.socket = gevent.socket
redis = Redis()

def enqueue(message):
    redis.lpush('work_queue', message)


while True:
    message = socket.recv()
    #print message
    gevent.spawn(enqueue, message)

If I remove gevent stuff, then the code works (messages are stored correctly).

P.S. I have just started to play with zeromq and gevent.

Update: I wanted to implement a simple task queue. In ended up using Celery. The server (which use zmq) asynchronously starts celery tasks (and it works great).

Upvotes: 2

Views: 2506

Answers (3)

kaka_ace
kaka_ace

Reputation: 387

with redis-py pyzmq, i write a demo, gevent version 1.0.1,

References:
(sorry, my reputation is less than 10, i could't post more than two 2 links):

[ gevent with redis-py ]
[ gevent with zmq ]

the reference links just show in the demo code :)

the demo works well. [ My demo code ]

Upvotes: 1

Mauricio Souza Lima
Mauricio Souza Lima

Reputation: 608

To use zeromq with gevent, you have to import de zmq.green module.

import zmq.green as zmq

http://zeromq.github.io/pyzmq/api/zmq.green.html

Upvotes: 2

The Real Bill
The Real Bill

Reputation: 15813

You don't appear to be starting the event loop. According to the tutorial at http://sdiehl.github.io/gevent-tutorial/ try wrapping the spawn call in a gevent.joinall call. Try that to see if it works then. Caution, that may not be the ideal way to use gevent. But it should be a start.

Upvotes: 1

Related Questions