CODEWITHSUNDEEP
pythonmemory
Merlin
Merlin

Reputation: 25679

Python Memory Error

I'm runing into Memory Error Using python 2.6, pyodbc

The code loops through several Sql statements. Singleton run Ok, but the loops get stuck. Sometimes Loop works ok. 

for Sql in LoopList:
     f = csr.execute(Sql)
     LL = list(f)

The second element in loop crashes. 

Sql:

Sql   =""" SELECT * FROM group WHERE 
        C>10 AND M <0 AND S<0  
        AND TC >= 200 AND OC >=1000  and Penny =.01 
        ORDER BY MSlp           
"""

Upvotes: 0

Views: 2007

Answers (3)

Asif Mehmood
Asif Mehmood

Reputation: 103

Since your query is returning too much data that it cannot be stored in RAM, for a quick solution try increasing your virtual memory.

  1. Open My Computer
  2. Right click and select Properties
  3. Go to Advanced System Settings
  4. Click on Advance Tab
  5. Click on settings under Performance
  6. Click on Change under Advance Tab
  7. Increase the memory size, that will increase virtual memory size.

It will convert some of your hard disk space to virtual memory, and your issue will be resolved now if your data fits into allocated memory.

Upvotes: 0

Merlin
Merlin

Reputation: 25679

The solution was to do sql looplist inside MySQL as Stored procedure, call from python. This eliminated the python error.

also use numpy for testing bytesize

Upvotes: 0

Mark Ransom
Mark Ransom

Reputation: 308520

Your query is returning too much data to fit into memory. The execute is probably returning an iterator which only needs to keep one item in memory at a time, but converting that into a list requires every single item returned from the query to fit into memory.

Upvotes: 4

Related Questions