CODEWITHSUNDEEP
pythonpostgresqlsqlalchemy
delalli
delalli

Reputation: 67

Running queries in parallel in sqlalchemy

I have a table (say table_A) in a db A, there are two other tables in db's B and C. I am running query like :

session.query(Table_A).filter(Table_A.date >= time1, InventoryLog.audit_date < time2)

I am running it in a loop, adding the rows in table_B and updating time1,time2 in each iteration.

Now, I want to run another query in parallel on the same (or different table) for different time intervals and insert in another table_C (loop). The important point is to run them parallely.

How should I go about it in sqlalchemy? Should I use celery for this or will that be an overkill?

Thanks in advance!

Upvotes: 0

Views: 3865

Answers (1)

talhasch
talhasch

Reputation: 367

You can use threading for paralel processing. 

from threading import Thread

def fn1(arg1):
    while True:
        session.query(Table_A).filter(Table_A.date >= time1, InventoryLog.audit_date < time2)
        # do other stuff


def fn2(arg1):
    while True:
        session.query(Table_B).filter(Table_B.date >= time1, InventoryLog.audit_date < time2)
        # do other stuff


def fn3(arg1):
    while True:
        session.query(Table_C).filter(Table_C.date >= time1, InventoryLog.audit_date < time2)
        # do other stuff


t1 = Thread(target=fn1, args=(1,))
t2 = Thread(target=fn2, args=(1,))
t3 = Thread(target=fn3, args=(1,))

t1.start()
t2.start()
t3.start()

Upvotes: 1

Related Questions