CODEWITHSUNDEEP
pythonsqlite
Cripto
Cripto

Reputation: 3751

Sqlite running out of memory on a python call

So, when I run my code, which is pulling a lot of data from a sqlite database,

I get the following error in my python code!

(<type 'exceptions.MemoryError'>, 'main.py', 427)

Yes, the above exception is some of my own error formatting.

Before it was just printing MemoryError

This is what I have on line 427:

rows=cur.fetchall()

Then I assumeed the problem has to do with memory: so I did the following

sqlite> PRAGMA page_size = 1073741824;
sqlite> PRAGMA cache_size = 100000;
sqlite> VACUUM;


sqlite> PRAGMA page_size;
4096
Memory Used:                         50186800 (max 100373936) bytes
Number of Outstanding Allocations:   12075 (max 24216)
Number of Pcache Overflow Bytes:     50045536 (max 100095168) bytes
Number of Scratch Overflow Bytes:    0 (max 11392) bytes
Largest Allocation:                  469214 bytes
Largest Pcache Allocation:           4244 bytes
Largest Scratch Allocation:          11392 bytes
Lookaside Slots Used:                0 (max 0)
Successful lookaside attempts:       0
Lookaside failures due to size:      0
Lookaside failures due to OOM:       0
Pager Heap Usage:                    49904696 bytes
Page cache hits:                     2011
Page cache misses:                   21561
Page cache writes:                   11780
Schema Heap Usage:                   7296 bytes
Statement Heap/Lookaside Usage:      1448 bytes
Fullscan Steps:                      0
Sort Operations:                     0
Autoindex Inserts:                   0
sqlite>


sqlite> .version
SQLite 3.7.13 2012-06-11 02:05:22

But I still get the same error.

Thoughts?

EDIT:

Per answer blow, attempted:

rows = []
while True:
   rows.append(cur.fetchone())

But the same result.

Upvotes: 0

Views: 1375

Answers (1)

CL.
CL.

Reputation: 180020

The size of your database file is not necessarily related with the size of all the records returned by a query.

fetchall must load all results into memory at once. You would be better off calling fetchone repeatedly and handling the records individually.

Upvotes: 2

Related Questions