CODEWITHSUNDEEP
pythondjango
blaz1988
blaz1988

Reputation: 366

Hot to optimize RAM usage in Django/Python?

I need help to reduce RAM usage of this python script:

tempproducts = TempProducts.objects.filter()

for url in tempproducts:
    scrap_and_save_product(url.url,True,0)

scrap_and_save_product is a function which scraps data with BeautifulSoup and saves it in database .. TempProducts has about 100,000 products

Every 30 minutes RAM usage increase by about 50 MB

RAM usage log :

220.059  MB 
271.594  MB 
313.316 MB
355.492 MB
373.516 MB
402.266 MB
437.328 MB
470.746 MB
507.195 MB
543.059 MB
574.215 MB
614.906 MB
643.902 MB
742.559 MB
787.93 MB
823.988 MB
856.949 MB
896.645 MB
931.93 MB
964.68 MB

How I can check exactly which data structure takes my RAM? How I can reduce RAM usage in Python/Django? ... Is better to use tuple or list?

Upvotes: 0

Views: 403

Answers (4)

Ned Batchelder
Ned Batchelder

Reputation: 375574

Make sure that you have DEBUG = False in your settings. If this setting is true, then SQL results are kept for debugging.

Upvotes: 2

koxt
koxt

Reputation: 693

Use iterator(). But don't forget it won't cache results.

For a QuerySet which returns a large number of objects that you only need to access once, this can result in better performance and a significant reduction in memory.

Upvotes: 1

Sudip Kafle
Sudip Kafle

Reputation: 4391

Use tempproducts=TempProducts.objects.filter().values('url')

From whatever you have shown in the code, you only require url from the database. Instead you are fetching everything that's stored in the table which is ultimately saved in your RAM.

The query: tempproducts=TempProducts.objects.filter() translates to something like:

SELECT * from tempproducts_table;

while temproducts = TemProducts.objects.filter().values('url',) translates to:

SELECT url from tempproducts_table;

Upvotes: 0

Albert Tugushev
Albert Tugushev

Reputation: 1712

  1. You should show us code of scrap_and_save_product function.
  2. Try to make more memory-efficient queries with large data. Details described here. Hope these helps!

Upvotes: 2

Related Questions