CODEWITHSUNDEEP
pythonpython-3.xsqlalchemypython-multiprocessing
Cristhian Boujon
Cristhian Boujon

Reputation: 4190

Handling scoped_session across multiple modules with SQLAlchemy

I'm newbie using SQLAlchemy and I'm working on a complex ETL process so I did the below simplified code:

module1.py

class Foo:
    def foo_method(self):
        # doing stuff with database

module2.py

class Bar:
    def bar_method(self):
        # doing stuff with database

main_script.py

from module1 import Foo
from module2 import Bar

def run():
    with Pool(processes = num_workers) as pool:
        responses = [pool.apply_async(some_func, (param)) for param in params]
        for response in responses:
            response.get()


def some_func(param):
    engine = create_engine(connection_string, echo=True)
    Session = scoped_session(sessionmaker(bind=engine))
    session = Session()    
    # Start doing some stuff with database
    foo = Foo()
    foo.foo_method()

    bar = Bar()
    bar.bar_method()

So I have a Pool with worker process. When I call main_script.run() each worker creates a database session inside some_func. My question is how can I use the same session for each worker in module1 and module2 without passing the session by param to each method? Should I add the follow lines in each module/file?

    engine = create_engine(connection_string, echo=True)
    Session = scoped_session(sessionmaker(bind=engine))
    session = Session()

Upvotes: 4

Views: 924

Answers (1)

univerio
univerio

Reputation: 20538

scoped_session should be created at the module level. For your project structure, that probably means having a separate module to house the engine and session:

db.py

engine = create_engine(connection_string, echo=True)
Session = scoped_session(sessionmaker(bind=engine))

module1.py

from db import Session

class Foo:
    def foo_method(self):
        session = Session()
        session.query(...)...

Upvotes: 3

Related Questions