CODEWITHSUNDEEP
pythonmultithreadingcelery
Matt
Matt

Reputation: 173

Property tied to a task instance in celery

Let's say I have:

I want to tie each opener to a Task instance (so each task has it's own opener, ie. with different auth cookies).

What I currently do is subclass from Task:

    class TaskWithOpener(Task):
        abstract = True
        _openers = None

        @property
        def openers(self):
            if self._openers is None:
                print 'creating openers for', self
                (...)
            print 'openers already created for ', self, ' just returning them'
            return self._openers

and make task like this:

    @my_celery.task(rate_limit='5/m', base=TaskWithOpener)
    def my_task():
        opener = random.choice(my_task.openers)

But this way each task has list of multiple openers and they are created for each thread separately so when there are 3 credential pairs (login, password) and concurrency = 3, my program creates 9 openers which is unacceptable.

Upvotes: 1

Views: 1189

Answers (1)

Krzysztof Szularz
Krzysztof Szularz

Reputation: 5249

This is perfectly valid behavior of Celery. You've basically created a class that for each instance creates three openers and instantiated it three times.

What you're trying to do is to spawn three tasks, each with its own set of credentials:

@celery.task(rate_limit='5/m')
def the_task(login, password):
    opener = create_opener(login, password)
    …

Then you can call it like:

credentials = [
    ('login1', 'password1'),
    ('login2', 'password2'),
    ('login3', 'password3'),
]

for login, password in credentials:
    the_task.delay(login, password)

That way the worker will receive three tasks and apply rate limit to it.

Update:

From your comment and the code I suspect you want to make options a class attribute.

The problem is overwriting the attribute on self makes it an instance attribute.

I think you're trying to create a class property.

Honestly, I don't think this is a good solution. I would like to know why you don't want to create the opener each time.

Is this costly? Then what you're looking for is not a task queue + worker but some server running constantly (may be implemented over twisted).

Upvotes: 1

Related Questions