CODEWITHSUNDEEP
phpgearman
Wizzard
Wizzard

Reputation: 12702

Gearman workers - limiting the amount of jobs a worker will process

Afternoon,

We are using Gearman in a production environment, we have a couple of workers running which are processing 100's of thousands of jobs daily (busy workers). We are having a slight memory issue though, the leak is very small, but after a few days it's building up.

On the command line, when I run 

gearman --help

It says

...
Worker options:
-c <count> - Number of jobs for worker to run before exiting

This seems like the ideal option, we could set it to 50~k and know that the worker will die off and problem is solved. However looking through the pecl library for gearman I can't see where I can add/configure that.

There is an AddOption method which takes an int - but I can't find any reference in the document or source code to where I can set this.

I don't want to configure my own management system...

Any ideas?

Upvotes: 2

Views: 3017

Answers (2)

Louis-Philippe Huberdeau
Louis-Philippe Huberdeau

Reputation: 5431

You should probably post some code about how you handle the workers, because adding such a limit is usually trivial. Based on the sample from the manual:

$worker = new GearmanWorker(); 
$worker->addServer(); 

$worker->addFunction("reverse", "my_reverse_function"); 

while ($worker->work());

The worker performs a single operation at a time, so all you need to do is replace the while loop for something else.

for ($i; 50000 > $i; ++$i) { $worker->work(); }

I don't know how you manage to keep the workers alive, but I have found Upstart to work nicely.

Upvotes: 1

MatsLindh
MatsLindh

Reputation: 52832

GearmanManager allows you to specify the time each worker should live before being restarted (in seconds, not performed tasks).

Otherwise you could just keep track of the number of performed tasks in your worker, and start it with a bash-script in a while-loop. When you reach 50k, just exit() and let the bash script restart your worker.

Upvotes: 1

Related Questions