CODEWITHSUNDEEP
pythonpostgresqlmultiprocessingpsycopg2
maverickk89
maverickk89

Reputation: 81

Multiprocess over python with psycopg2

I'm working on python to add many records into a Postgres DB, because I have to insert millions records I need to parallelize 2 functions:

insert_A(cur)  
insert_B(cur)

where:

conn=psycopg2.connect(.....)
cur = conn.cursor()

I tried a solution I found in another post here in stack overflow like: 

result1= pool.apply_async(insert_A, cur)  
result2= pool.apply_async(insert_B, cur)  
answer1=result1.get(timeout=15)  
answer2=result2.get(timeout=15)

but I got this error: Traceback (most recent call last):
File "test.py", line 387, in
answer1=result1.get(timeout=15)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
psycopg2.InterfaceError: the cursor has no connection

somebody could help me please? :(

Upvotes: 2

Views: 10191

Answers (2)

maverickk89
maverickk89

Reputation: 81

in the end I solved in this way:

if __name__ == '__main__':
    p = Process(target=insert_A, args=(ARG,))
    q = Process(target=insert_measurement_real, args=(ARG,))
    p.start()
    q.start()
    p.join()
    q.join()

and creating the connection inside the function instead of creating it outside the function (and passing the connection as parameter)...

Upvotes: 5

fedterzi
fedterzi

Reputation: 1164

As this issue reports, it is not possible on psycopg2 to use a connection on a separate process, the connection must be established in the working process.

Upvotes: 5

Related Questions