CODEWITHSUNDEEP
pythonflasksqlalchemymultiprocessingflask-sqlalchemy
spketre
spketre

Reputation: 81

What is the correct way of using Flask-Sqlalchemy with multiprocessing?

I have a route in my Flask app that spawns a process (using multiprocessing.Process) to do some background work. That process needs to be able to write to the database.

__init__.py:

from flask_sqlalchemy import SQLAlchemy
from project.config import Config

db = SQLAlchemy()

# app factory
def create_app(config_class=Config):
    app = Flask(__name__)
    app.config.from_object(Config)
    db.init_app(app)
    return app

And this is the relevant code that illustrates that way i'm spawning the process and using the db connection:

def worker(row_id):    
    db_session = db.create_scoped_session()

    # Do stuff with db_session here

    db_session.close()

@app.route('/worker/<row_id>/start')
def start(row_id):
    p = Process(target=worker, args=(row_id,))
    p.start()
    return redirect('/')

The problem is that sometimes (not always) i have errors like this one:

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) insufficient data in "D" message lost synchronization with server: got message type "a", length 1668573551

I assume that this is related to the fact that there is another process accessing the database (because if i don't use a separate process, everything works fine) but i honestly can't find a way of fixing it. As you can see on my code, i tried used create_scoped_session() method as an attempt to fix this but the problem is the same.

Any help?

Upvotes: 4

Views: 1286

Answers (1)

spketre
spketre

Reputation: 81

Ok so, i followed @antont 's hint and created a new sqlalchemy session inside the worker function this way and it worked flawlessly:

from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

def worker(row_id):    
    db_url = os.environ['DATABASE_URL']
    db_engine = create_engine(db_url)
    Session = sessionmaker(bind=db_engine)
    db_session = Session()

    # Do stuff with db_session here

    db_session.close()

Upvotes: 3

Related Questions