CODEWITHSUNDEEP
pythonmemoryurllib2mechanize
Shane
Shane

Reputation: 4983

"Out of Memory" error with mechanize

I was trying to scrape some information from a website page by page, basically here's what I did: 

import mechanize
MechBrowser = mechanize.Browser()

Counter = 0

while Counter < 5000:
    Response = MechBrowser.open("http://example.com/page" + str(Counter))
    Html = Response.read()
    Response.close()

    OutputFile = open("Output.txt", "a")
    OutputFile.write(Html)
    OutputFile.close()

    Counter = Counter + 1

Well, the above codes ended up throwing out "Out of Memory" error and in task manager it shows that the script used up almost 1GB memory after several hours running... how come?!

Would anybody tell me what went wrong?

Upvotes: 5

Views: 1251

Answers (1)

Josh
Josh

Reputation: 2196

This is not exactly a memory leak, but rather an undocumented feature. Basically, mechanize.Browser() is collectively storing all browser history in memory as it goes.

If you add a call to MechBrowser.clear_history() after Response.close(), it should resolve the problem.

Upvotes: 14

Related Questions