CODEWITHSUNDEEP
pythonpostgresqlasynchronousmultiprocessingpgdb
Antonin
Antonin

Reputation: 1830

pg.InternalError: SSL SYSCALL error: EOF detected

I have a code in python using multiprocessing and accessing a database. It works most of the time properly but from time to time it creates an error. The error was originally 

File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
    raise self._value
pg.OperationalError: can't rollback

or 

File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
    raise self._value 
pg.InternalError: SSL SYSCALL error: EOF detected

The code I use to access the data base is

def getRecords(db,query):
    cur=db.cursor()
    cur.execute(query)
    types=getTypes(cur.description)
    columns=getColumns(cur.description)
    rows=cur.fetchall()
    cur.close()
    try:
        db.commit()
    except:
        db.rollback()

Multiprocessing is done through:

po = multiprocessing.Pool()
for an_element in a_list:
    for an_object in a_list_of_objects:
        results.append(po.apply_async(the_multiprocessed_function, (an_element, an_object)))               
po.close()
po.join()
for r in results:
    some_function_of(r.get())

I was wondering if I was sharing connections across processes, but for each multiprocessed function, I create I new connection (conn = pgdb.connect(...)) and I close it (conn.close()).

Then, I was wondering if it was related to the parameters I'm giving to my multiprocessed function,in particular because they are class instances I defined myself, but why would this create an error in pg?

Sorry if my question is not very clear, but I don't know where to look for the mistake. The multiprocessing worked very well with smaller instances and other tables and has been tested. But when changing one table in the query and running the code for bigger problems, I reach some limit somewhere.

Upvotes: 4

Views: 11530

Answers (1)

Antonin
Antonin

Reputation: 1830

In fact, this error was not related to multiprocessing but just to a function in postgresql/pgrouting that compute the shortest path. When no path connects two points in a network, the function disconnect from the database, with no explanation. See psycopg2.InterfaceError: connection already closed / pgr_astar for more about this.

Upvotes: 5

Related Questions