CODEWITHSUNDEEP
pythonmemory-managementmemory-leakssqlalchemymemory-consumption
Gosha null
Gosha null

Reputation: 641

SqlAlchemy memory management

I am trying to optimize memory consumption in one of the scripts and use memory_profiler module to track memory consumption.

Here's an example of what it shows:

 91.3 MiB      0.0 MiB           1           items = Items.query.filter(
106.1 MiB     14.9 MiB           1               Items.instance_id == instance_id,
                                             ).all()
106.1 MiB      0.0 MiB        4116           items_ids = [i.id for i in items]
106.1 MiB      0.0 MiB           1           zipfile.append(
106.1 MiB      0.0 MiB           1               "items.json",
106.1 MiB      0.0 MiB           1               json.dumps(
108.4 MiB      2.3 MiB           1                   ItemDumpSchema(many=True).dump(items)
                                                 )
                                             )
106.9 MiB     -1.5 MiB           1           del items

As you can see, when getting a list of elements from the database, memory consumption increased by 14.9MiB.

Then I save this data to a zip archive in RAM and the compressed data in it increases the consumption by another 2.3 MiB.

But when I call del items after that, the memory is freed by only 1.5 MiB.

I think sqlalchemy still keeps references to these objects as they are bound to the current session, but I don't understand how to clear them.

I tried session.expunge_all () but it didn't work. Also I tried calling gc.collect() and that didn't work either.

SQLAlchemy version 1.2.12

Python version 3.6

Upvotes: 1

Views: 707

Answers (0)

Related Questions