CODEWITHSUNDEEP
pythonsqlitesqlalchemy
ABC
ABC

Reputation: 399

Multiprocessing and SQLAlchemy Error

We are building a software that reads thousands of XML files and imports their content into a SQLite database using SQLAlchemy.

Please find below the relevant parts of our code:

File 1:

from multiprocessing.dummy import Pool as ThreadPool
...
for file in os.listdir(path):
    if file.endswith("-endpoint.xml"):
        xmls.append([self.database, os.path.join(path, file), self.debug])

pool = ThreadPool(threads)
try:
    def multi_run_wrapper(args):
        return collectors.import_asset(*args)

    results = pool.map(multi_run_wrapper, xmls)
    pool.close()
    pool.join()
except:
    if self.debug:
        traceback.print_exc(file=sys.stdout)

File 2, part 1:

def import_asset(db, xml, debug = False):
    tree = ET.parse(xml)
    root = tree.getroot()
    signcheck = root.find('Signcheck')
    usersArray = []
    running_processes = []
    autoruns = []

    machine_id = add_machine(root, xml, db, debug)

File 2, part 2:

def add_machine(root, file_path, db, debug = False):
    ...
    machine = Machine(hostname=hostname, operating_system=operating_system, version=Version)
    db.add(machine)
    db.commit()

File 3:

Throughout the code, 'db' is whatever the return value of this function is:

def open(sqlite_db_path = None):
    engine = create_engine('sqlite:///' + sqlite_db_path)
    session = sessionmaker()
    session.configure(bind=engine)
    return scoped_session(session())

Error Message:

TypeError: 'Session' object is not callable

We have read here that scoped_session() does not return a true Session() object. But we have not been able to use that information to fix our code.

Thank you in advance for your kind assistance with this hurdle.

Upvotes: 0

Views: 2412

Answers (2)

ABC
ABC

Reputation: 399

The fix for this was simply to replace this:

def open(sqlite_db_path = None):
    engine = create_engine('sqlite:///' + sqlite_db_path)
    session = sessionmaker()
    session.configure(bind=engine)
    return scoped_session(session())

With this:

engine = create_engine('sqlite:///' + sqlite_db_path)
session_factory = sessionmaker(bind=engine)
return scoped_session(session_factory)

Thanks to all the people that responded to this thread.

Upvotes: 3

Lukas Graf
Lukas Graf

Reputation: 32590

You're calling the session factory session, when you should be calling the scoped_session instead (see Contextual/Thread-local Sessions).

So it should be

scoped_session(session)()

instead of 

scoped_session(session())

In order to avoid confusion, I would recommend to rename session to session_factory. So something like this should work:

def open(sqlite_db_path = None):
    engine = create_engine('sqlite:///' + sqlite_db_path)
    session_factory = sessionmaker(bind=engine)
    return scoped_session(session_factory)()

Upvotes: 4

Related Questions