CODEWITHSUNDEEP
pythonmultithreadingorientdb
kenlz
kenlz

Reputation: 461

Multithreading Python For DB Insert

Hi guys currently i am trying to migrate current db to another and that process unfortunately involves python. I was able to do it single threaded, but it's incredibly slow took hours to finish 1M data. Is there similar method in python like Java executor and futures?

note that user_list is a chunk of 1000/1M

for data in user_list:
    q = """ insert into users(id,name,address,password) 
        Values({id},{name},{address},{password})
        """.format(id=data['id'],name=data['name'],address=data['address'],password=data['password'])
    db.command(q)

I think it would be a whole lot faster if i run for example 8 concurrent threads inserting 8 at a time instead of single thread doing single insert

Upvotes: 0

Views: 3517

Answers (1)

ErikR
ErikR

Reputation: 52039

Since you say in the comments that you are using orientdb, have a look at the SQL Batch capability.

Using SQL BATCH does not insert rows in parallel, but it will avoid the round-trip for each command.

You can also use SQL BATCH from Python using the pyorient library:

https://github.com/mogui/pyorient#execute-orientdb-sql-batch

To insert data in parallel you will need to create multiple connections, one for each thread.

Upvotes: 1

Related Questions