CODEWITHSUNDEEP
pythonmysqlcelerycelery-task
Rajs123
Rajs123

Reputation: 663

How to enter arguments of celery jobs into database to be queried; using mysql

I am using MySql as result backend for celery. I want to store program arguments in DB too.

For example: 

add.apply_async(args=[num1, num2, user]

In this case I want to store user argument in DB, so that I can query later.

Currently, I return arguments provided which is stored in DB.

def add(num1, num2, user):
    return [num1+num2, user]

However when the task is running state, user is not inserted and I am unable to query it in DB. Is there any solution/hack for this?

Upvotes: 10

Views: 1038

Answers (3)

blueboy
blueboy

Reputation: 64

You need queue,each user build a queue for himself, celery tasks get arguments from the user queue.

queue.get(timeout=10)# if nothing got then retry util get the arguments

Upvotes: 0

user305883
user305883

Reputation: 1741

Assuming you are able to connect and select to the db, you may want to separate a function that query parameters and a function callback to store your results in db.

Add() will operate results, and store_callback() will store them in db when ready. So if parameters are ready, your code can go on next task and store result (user) when done.

Something like:

def store_callback(result):
     sql_insert = 'INSERT INTO your_table VALUES(?, ?,)'
     curs.execute(sql_insert, result)  #result is a list passed from add()

def add(num1, num2, user):
    return [num1+num2, user]


# check parameters are present in db:
curs.execute("SELECT * FROM your_table WHERE user = ?", [_user])
user_exists = curs.fetchone()
#
if user_exists:
    add.apply_async( (num1, num2, user, ) , link=store_callback.s() )

you can even link add() to another task.

Upvotes: 0

Seán Hayes
Seán Hayes

Reputation: 4360

Are you only using MySql as a result backend, or are you using it for the queue as well (which I wouldn't recommend)? If you're using it for a queue, the args should be in the database as soon as the task has been sent. Otherwise, the task result can't be stored until after the task has finished.

If you need the arguments to be query-able while the task is running, you'll need to manually insert them into a DB table at the start of your task. If you want them to be query-able before the task starts, you'll need to insert them into a DB table right before calling apply_async.

Upvotes: 1

Related Questions