CODEWITHSUNDEEP
pythonpython-2.7seleniumpython-3.xselenium-webdriver
user3810188
user3810188

Reputation: 311

Python - Selenium webdriver error after running for few hours

I'm running a BOT in cloud machine(AWS 1GB instance), I'm using selenium-webdriver + pyvirtualdisplay.

I'm doing some repetitive stuff in a loop which will iterate 50000 times. after running my script it stopped working at 1016th (out of 50000). and error was 

Traceback (most recent call last):
  File "automation.py", line 108, in <module>
    main()
  File "automation.py", line 10, in main
    driver = webdriver.Firefox()
  File "/home/eric/work/lak/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
    self.binary, timeout),
  File "/home/eric/work/lak/local/lib/python2.7/site-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
    self.binary.launch_browser(self.profile)
  File "/home/eric/work/lak/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 61, in launch_browser
    self._wait_until_connectable()
  File "/home/eric/work/lak/local/lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 105, in _wait_until_connectable
    self.profile.path, self._get_firefox_output()))
selenium.common.exceptions.WebDriverException: Message: 'Can\'t load the profile. Profile Dir: /tmp/tmp7CXXn1 Firefox output: \n(process:22693): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\nXlib:  extension "RANDR" missing on display ":1213".\n1406467599310\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]\n1406467599312\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/LightweightThemeManager.jsm: ["LightweightThemeManager"]\n1406467599314\taddons.xpi\tDEBUG\tstartup\n1406467599339\taddons.xpi\tDEBUG\tcheckForChanges\n1406467599351\taddons.xpi\tDEBUG\tNo changes found\nSystem JS : ERROR chrome://browser/content/browser.js:12132 - SyntaxError: function statement requires a name\n 
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n
System JS : ERROR (null):0 - out of memory\n'

From error It seems that there was some memory issue? was it? I've no clue how to fix it?

Infact If I re-run this script it doesn't, it shows this error.

selenium.common.exceptions.WebDriverException: Message: 'The browser appears to have exited before we could connect. The output was: \n(process:22882): GLib-CRITICAL **: g_slice_set_config: assertion \'sys_page_size == 0\' failed\nXlib:  extension "RANDR" missing on display ":1295".\n1406470964822\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]\n1406470964823\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/LightweightThemeManager.jsm: ["LightweightThemeManager"]\n1406470964826\taddons.xpi\tDEBUG\tstartup\n1406470964847\taddons.xpi\tDEBUG\tcheckForChanges\n1406470964855\taddons.xpi\tDEBUG\tNo changes found\n************************************************************\n* Call to xpconnect wrapped JSObject produced this error:  *\n[Exception... "Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]"  nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)"  location: "JS frame :: file:///tmp/tmpdCliUY/extensions/[email protected]/components/bad_cert_listener.js :: WdCertOverrideService :: line 3855"  data: no]\n************************************************************\n************************************************************\n* Call to xpconnect wrapped JSObject produced this error:  *\n[Exception... "Component returned failure code: 0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE) [nsIJSCID.getService]"  nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)"  location: "JS frame :: file:///tmp/tmpdCliUY/extensions/[email protected]/components/bad_cert_listener.js :: WdCertOverrideService :: line 3855"  data: no]\n************************************************************\nJavaScript error: chrome://fxdriver/content/server.js, line 38: NS_ERROR_XPC_CI_RETURNED_FAILURE: Component returned failure code: 0x80570015 (NS_ERROR_XPC_CI_RETURNED_FAILURE) [nsIJSCID.createInstance]\n\n(firefox:22882): LIBDBUSMENU-GLIB-WARNING **: Unable to get session bus: Failed to fork (Cannot allocate memory)\nout of memory: 0x0000000000040028 bytes requested\n'

For your reference here is my code looks like.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from pyvirtualdisplay import Display

display = Display(visible=0, size=(1024, 768))
display.start()
driver = webdriver.Firefox()
driver.maximize_window()

url = 'some url'

for i in range(1,50000):
    driver.get(url)
    #somestuff
    i+=1

Upvotes: 0

Views: 1784

Answers (1)

Gaurav Jain
Gaurav Jain

Reputation: 1865

from error logs it seems you are using unix flavor

I had same kind of error in past. Clearly we can see system is unable to provide enough memory.

run top command and see if free memory is enough. if not then kill some unnecessary processes.

This should work.

Upvotes: 1

Related Questions