CODEWITHSUNDEEP
pythonpsycopg2multiprocess
user2045508
user2045508

Reputation: 43

Multi-threaded psycopg2 and python not returning results

I have a program where the parent process has a DB connection, and each child process has its own DB connection (created in the constructor), using python 2.6 and psycopg2.

Every 5 seconds, the parent process queries the DB to get a progress report on the child process. Each child process is doing X things and storing what step it is on in the DB.

I've put a simplified version of the code below

def getStatus( conn ):
    query = "select job_name, status from job_status_table"
    cursor = conn.getCursor()
    cursor.execute( query )
    return cursor.fetchAll()


def simpleStatus():
    conn = DBInit()
    # output is correct here
    print getStatus( conn )
    queue = getJobList( conn )
    for q in queue:
        p = multiprocessing.Process( target=run, args=q )
        p.start()
    while: # condition that does terminate, not germane
        time.sleep( 5 )
        # output is incorrect here
        print getStatus( conn )
    ...

Inside the child process, it calls the following:

def updateStatus( self, status_i ):
    update = "update job_status_table set status='%s' where job_name='%s'"%( status_i, self.name )
    cursor = self.conn.getCursor()
    cursor.execute( update )
    self.conn.commit()

External querying of the database (psql) shows that the query is returning the correct results at the time that it is being run in the program. However, in the program it is not. If I change the program to re-initialize the DB after the time.sleep call, it gives the correct output. Why?

Upvotes: 4

Views: 1148

Answers (1)

fog
fog

Reputation: 3391

The parent process is in its own transaction and will not see any changes until it terminates it (by commit() or rollback()). You have two choices:

  1. put the parent process connection in autocommit mode (conn.autocommit = True); or
  2. just issue a commit()/rollback() on the connection before executing your query to make sure to execute it in a new, up to date, transaction.

Upvotes: 3

Related Questions