sqlalchemy flush() and get inserted id?

Question

I want to do something like this:

f = Foo(bar='x')
session.add(f)
session.flush()

# do additional queries using f.id before commit()
print f.id # should be not None

session.commit()

But f.id is None when I try it. How can I get this to work?

Answer 1

I was running into the same problem of missing id values after creating objects whilst using bulk_save_objects.

Luckily the return_defaults parameter in bulk_save_objects adds all the ids to the objects without having to call flush or commit.

users: list[User] = [User(user_name=..., email=..., etc) for foo in bar]
session.bulk_save_objects(references, return_defaults=True)
assert users[0].id is not None # User.id is now populated.

See SQLAlchemy documentation: :param return_defaults: when True, rows that are missing values which generate defaults, namely integer primary key defaults and sequences, will be inserted one at a time, so that the primary key value is available.

Answer 2

The last couple of hours/days/whatever, I was trying to get the above suggestions to work. Initially, I wrote all my insert functions like so:

_add = User(id, user_name, email, ...)

Where all the items between the round brackets are variables for None, "user a", "a@example.com", ...

This is my User table:

class User(Base):
    __tablename__ = "users"
    
    id = Column(Integer, primary_key=True, autoincrement=True)
    user_name = Column(String(50), unique=True, nullable=False)
    email = Column(String(100), unique=True, nullable=False)

SQLAlchemy handles the _add query correctly, as it inserts the record with an autoincremented ID. Also, as should be, no default value is set for the id column.

I have tried all the above options in various ways (with/without commit, with/without flush, with/without refresh, the one before the other, timeouts in between statements, you name it). I even changed the whole app/database interaction a couple of times. But in all occasions, "_add.id" would either return 0, or something like "Instance '' has been deleted, or its row is otherwise not present."

Just now I thought "maybe I should write my _add query a bit different, by also defining the column names for the specified table" like so:

_add = User(id=None, user_name=user_name, email=email, etc)

To emphasize, note: id=, user_name=, email=, in the _add query. Now, with the following statements in this order, SQLAlchemy does return the inserted ID!

session.add(_add)
print(_add.id)    <-- returns None

session.flush()   <-- does **not** insert record into database, but does increment id,
                      waiting to be committed. Flush may be omitted, because
                      session.commit() unconditionally issues session.flush()*
print(_add.id)    <-- returns incremented id

session.commit()  <-- commit is needed to actually insert record into database
print(_add.id)    <-- returns incremented id

Although the answer has been provided, it wasn't clear for me the missing column names in the _add query, and thus my laziness, were the cause of my problems. I hope this can help someone avoid the same troubleshoot...

SQLAlchemy docs

Answer 3

The core solution has been mentioned in other much older answers, but this uses newer async API.

with sqlalchemy==1.4 (2.0 style), following seems to work:

from sqlalchemy.ext.asyncio import AsyncSession
from sqlalchemy.ext.asyncio import create_async_engine

engine = create_async_engine(
        "postgresql+asyncpg://user:pass@localhost/db",
        echo=False,
    )


# expire_on_commit=False will prevent attributes from being expired
# after commit.
async_session = sessionmaker(
    engine, expire_on_commit=False, class_=AsyncSession,
)
# default kwarg autoflush=True


async with async_session() as session: 
    async with session.begin(): 
        f = Foo(bar='x')
        session.add(f)
        print(f.id)
        # None

        await session.flush()
        print(f.id)
        # not None
    # commits transaction, closes session

Answer 4

my code works like that:

f = Foo(bar="blabla")
session.add(f)
session.flush()
session.refresh(f, attribute_names=[columns name that you want retrieve]
# so now you can access the id inserted, for example
return f.id # id inserted will be returned

Answer 5

Your sample code should have worked as it is. SQLAlchemy should be providing a value for f.id, assuming its an autogenerating primary-key column. Primary-key attributes are populated immediately within the flush() process as they are generated, and no call to commit() should be required. So the answer here lies in one or more of the following:

1. The details of your mapping
2. If there are any odd quirks of the backend in use (such as, SQLite doesn't generate integer values for a composite primary key)
3. What the emitted SQL says when you turn on echo

Answer 6

Thanks for everybody. I solved my problem by modifying the column mapping. For me, autoincrement=True is required.

origin:

id = Column('ID', Integer, primary_key=True, nullable=False)

after modified:

id = Column('ID', Integer, primary_key=True, autoincrement=True, nullable=True)

then

session.flush()  
print(f.id)

is ok!

Answer 7

I once had a problem with having assigned 0 to id before calling session.add method. The id was correctly assigned by the database but the correct id was not retrieved from the session after session.flush().

Answer 8

unlike the answer given by dpb, a refresh is not necessary. once you flush, you can access the id field, sqlalchemy automatically refreshes the id which is auto generated at the backend

I encountered this problem and figured the exact reason after some investigation, my model was created with id as integerfield and in my form the id was represented with hiddenfield( since i did not wanted to show the id in my form). The hidden field is by default represented as a text. once I changed the form to integerfield with widget=hiddenInput()) the problem was solved.

Answer 9

I've just run across the same problem, and after testing I have found that NONE of these answers are sufficient.

Currently, or as of sqlalchemy .6+, there is a very simple solution (I don't know if this exists in prior version, though I imagine it does):

session.refresh()

So, your code would look something like this:

f = Foo(bar=x)
session.add(f)
session.flush()
# At this point, the object f has been pushed to the DB, 
# and has been automatically assigned a unique primary key id

f.id
# is None

session.refresh(f)
# refresh updates given object in the session with its state in the DB
# (and can also only refresh certain attributes - search for documentation)

f.id
# is the automatically assigned primary key ID given in the database.

That's how to do it.

Answer 10

You should try using session.save_or_update(f) instead of session.add(f).