CODEWITHSUNDEEP
pythonselenium-webdrivercron
Captain_Meow_Meow
Captain_Meow_Meow

Reputation: 2541

Execute Python (selenium) script in crontab

I have read most of the python/cron here in stackoverflow and yet couldn't make my script run. I understood that I need to run my script through shell (using zsh & ipython by the way), but really I have no idea what to do :/

My simple code:

In the crontab-

*/1 * * * * ipython /home/usr/Data/progs/cron_test.py

My python script-

import pickle
    from selenium import webdriver

driver = webdriver.Firefox()
driver.get('http://www.google.com')

t=driver.current_url
pickle.dump(t,open('noreal','wb'))

I have tried some things already but to no avail: 

#!python ../python etc
SHELL = /usr/bin/zsh
PATH =/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

Simply because I don't know what the problem is, I assume something with running python script via interpreter, but I don't know what I am doing really :)

A working solution would be great, and I would really appreciate an explanation to accompany any solution so I could understand the why and how and not just be like 'It works! Thanks! Bye!'

Thanks for any help!

Update So far I have narrowed down the problem, and now python runs with the following settings:

*/3 * * * * /usr/local/bin/ipython /home/user/Data/progs/RF/cron_test.py

and I got a traceback:

[1;31m---------------------------------------------------------------------------[0m
[1;31mWebDriverException[0m                        Traceback (most recent call last)
[1;32m/usr/local/lib/python2.7/dist-packages/IPython/utils/py3compat.pyc[0m in [0;36mexecfile[1;34m(fname, *where)[0m
[0;32m    176[0m             [1;32melse[0m[1;33m:[0m[1;33m[0m[0m
[0;32m    177[0m                 [0mfilename[0m [1;33m=[0m [0mfname[0m[1;33m[0m[0m
[1;32m--> 178[1;33m             [0m__builtin__[0m[1;33m.[0m[0mexecfile[0m[1;33m([0m[0mfilename[0m[1;33m,[0m [1;33m*[0m[0mwhere[0m[1;33m)[0m[1;33m[0m[0m
[0m
[1;32m/home/user/Data/progs/FB/cron_test.py[0m in [0;36m<module>[1;34m()[0m
[0;32m      9[0m [1;33m[0m[0m
[0;32m     10[0m [1;33m[0m[0m
[1;32m---> 11[1;33m [0mdriver[0m [1;33m=[0m [0mwebdriver[0m[1;33m.[0m[0mFirefox[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m
[0m[0;32m     12[0m [0mdriver[0m[1;33m.[0m[0mget[0m[1;33m([0m[1;34m'http://www.google.com'[0m[1;33m)[0m[1;33m[0m[0m
[0;32m     13[0m [1;33m[0m[0m

[1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.pyc[0m in [0;36m__init__[1;34m(self, firefox_profile, firefox_binary, timeout, capabilities, proxy)[0m
[0;32m     58[0m         RemoteWebDriver.__init__(self,
[0;32m     59[0m             command_executor=ExtensionConnection("127.0.0.1", self.profile,
[1;32m---> 60[1;33m             self.binary, timeout),
[0m[0;32m     61[0m             desired_capabilities=capabilities)
[0;32m     62[0m         [0mself[0m[1;33m.[0m[0m_is_remote[0m [1;33m=[0m [0mFalse[0m[1;33m[0m[0m

[1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.pyc[0m in [0;36m__init__[1;34m(self, host, firefox_profile, firefox_binary, timeout)[0m
[0;32m     45[0m         [0mself[0m[1;33m.[0m[0mprofile[0m[1;33m.[0m[0madd_extension[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m
[0;32m     46[0m [1;33m[0m[0m
[1;32m---> 47[1;33m         [0mself[0m[1;33m.[0m[0mbinary[0m[1;33m.[0m[0mlaunch_browser[0m[1;33m([0m[0mself[0m[1;33m.[0m[0mprofile[0m[1;33m)[0m[1;33m[0m[0m
[0m[0;32m     48[0m         [0m_URL[0m [1;33m=[0m [1;34m"http://%s:%d/hub"[0m [1;33m%[0m [1;33m([0m[0mHOST[0m[1;33m,[0m [0mPORT[0m[1;33m)[0m[1;33m[0m[0m
[0;32m     49[0m         RemoteConnection.__init__(

[1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.pyc[0m in [0;36mlaunch_browser[1;34m(self, profile)[0m
[0;32m     59[0m [1;33m[0m[0m
[0;32m     60[0m         [0mself[0m[1;33m.[0m[0m_start_from_profile_path[0m[1;33m([0m[0mself[0m[1;33m.[0m[0mprofile[0m[1;33m.[0m[0mpath[0m[1;33m)[0m[1;33m[0m[0m
[1;32m---> 61[1;33m         [0mself[0m[1;33m.[0m[0m_wait_until_connectable[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m
[0m[0;32m     62[0m [1;33m[0m[0m
[0;32m     63[0m     [1;32mdef[0m [0mkill[0m[1;33m([0m[0mself[0m[1;33m)[0m[1;33m:[0m[1;33m[0m[0m

[1;32m/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.pyc[0m in [0;36m_wait_until_connectable[1;34m(self)[0m
[0;32m     98[0m                 raise WebDriverException("The browser appears to have exited "
[0;32m     99[0m                       [1;34m"before we could connect. The output was: %s"[0m [1;33m%[0m[1;33m[0m[0m
[1;32m--> 100[1;33m                       self._get_firefox_output())
[0m[0;32m    101[0m             [1;32mif[0m [0mcount[0m [1;33m==[0m [1;36m30[0m[1;33m:[0m[1;33m[0m[0m
[0;32m    102[0m                 [0mself[0m[1;33m.[0m[0mkill[0m[1;33m([0m[1;33m)[0m[1;33m[0m[0m

[1;31mWebDriverException[0m: Message: 'The browser appears to have exited before we could connect. The output was: Error: no display specified\n'

Upvotes: 7

Views: 15756

Answers (5)

oz123
oz123

Reputation: 28868

A quick guess, your job fails to run because it requires an X session. You should setup your test script to run in a headless mode.

update:
Your trace says exactly what I meant, firefox does not run without X, or a display if you want.

 The output was: Error: no display specified\n'

Save yourself more trouble, read the link I posted.

Upvotes: 4

Amit Ghosh
Amit Ghosh

Reputation: 1606

This issue has bugged me for near 6 hours.

So if I write /usr/bin/python3 -u /root/apps/bridge/cinkselenium.py in terminal it is working fine.

But it doesn't work for crontab. Right?

Adding

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()

will not help.

What you need is to make a shell script and write the following -

#!/usr/bin/env bash

export DISPLAY=:0
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
nohup /usr/bin/python3 -u /root/apps/bridge/cinkselenium.py > /root/apps/bridge/cinkselenium.log &y

Now run the shell script from Crontab -

* * * * * /root/apps/bridge/cinkselenium.sh

This is what works properly for selenium in crontab without headache in DigitalOcean servers. The paths will be different based on your server and program location.

Upvotes: 2

Tom&#225;š Marn&#253;
Tom&#225;š Marn&#253;

Reputation: 61

You can do it without crontab.

I found schedule modul: GIThub/schedule

Now your code should look like:

import schedule
import time
def job():
    import pickle
    from selenium import webdriver
    driver = webdriver.Firefox()
    driver.get('http://www.google.com')
    t=driver.current_url
    pickle.dump(t,open('noreal','wb'))
schedule.every(1).minutes.do(job)
while True:
    schedule.run_pending()
    time.sleep(1)

It is not best solution but it works.

Upvotes: 2

Rafayet Ullah
Rafayet Ullah

Reputation: 1158

Install Xvfb or Xephyr or Xvnc. Also install pyvirtualdisplay python module. This is for Ubuntu 14.04 and tested on 16.04 as well.

sudo apt-get install python-pip
sudo apt-get install xvfb xserver-xephyr vnc4server
sudo pip install pyvirtualdisplay
# optional
sudo apt-get install python-pil scrot
sudo pip install pyscreenshot

Now you can start Xvfb from your python script. It will start Xvfb when your script will be executed. Add the following lines to your code.

from pyvirtualdisplay import Display
display = Display(visible=0, size=(1024, 768))
display.start()

Here is the example code.

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(1024, 768))
display.start()
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()

display.stop()

Now you can mention the program in your crontab. But add the path of ipython instead of ipython only.

0 * * * * /usr/bin/python /home/usr/Data/progs/cron_test.py build

Upvotes: -1

Gid
Gid

Reputation: 383

As seen here, you need to give cron a display:

30 5 * * * export DISPLAY=:0; /home/me/good_morning.sh

Upvotes: 17

Related Questions