CODEWITHSUNDEEP
pythonqueuegevent
echo12345
echo12345

Reputation: 21

Gevent queue high CPU usage

Why this code permanently uses 95% of CPU? Producer doesn't sending any request, but cpu is hot. How to lower cpu usage without adding explicit non-zero sleep timeout in consumer greenlet?

from gevent import queue
from gevent import sleep
from gevent import Greenlet

def cons(q):
    while True:
        try:
            data = q.get_nowait()
        except queue.Empty:
            sleep(0)
            continue
        print data

def prod(q):
    while True:
        #q.put_nowait('hello')
        sleep(1)


if __name__ == '__main__':

    q = queue.Queue()
    cons1 = Greenlet.spawn(cons, q)
    prod1 = Greenlet.spawn(prod, q)
    cons1.join()
    prod1.join()

Upvotes: 2

Views: 2582

Answers (2)

echo12345
echo12345

Reputation: 21

Best solution of all recommended: Use queue.get blocking call inside separate greenlets for each queue.

Next part of code polls two gevent.queue queues + zmq.green socket inside greenlets inside greenlet =) CPU Load is near zero!

def _zmq_poll(self):
    while True:
        sockets = super(Core, self)._zmq_poll()
        if sockets:
            if sockets.get(self.z_ctr) == zmq.POLLIN:
                self.ctr_process()

def ctr_process(self):
    self.send_log('new CTR: %s')

def req_process(self):
    while True:
        req = self.q_req_core.get()
        self.send_log('new req: %s' % req)
        self.q_core_wrk.put(req)

def wrk_process(self):
    while True:
        wrk_result = self.q_wrk_core.get()
        self.send_log('new result: %s' % wrk_result)
        self.q_core_res.put(wrk_result)
        self.test_wrk_result(wrk_result)

def _run(self):
    self.slaves_start()
    req_process = Greenlet(self.req_process)
    wrk_process = Greenlet(self.wrk_process)
    zmq_poll = Greenlet(self._zmq_poll)
    req_process.start()
    wrk_process.start()
    zmq_poll.start()
    zmq_poll.join()
    req_process.join()
    wrk_process.join()
    self._cleanup()

Thanks for advices!

Upvotes: 0

mogul
mogul

Reputation: 4553

In line 8, use get() instead of get_nowait()

data = q.get()

Super simple, now your consumer will block while waiting for more work.

Upvotes: 1

Related Questions