How can I use chromedriver(python) in AWS ec2?

Question

I need a little help on my little project on learning python web scraping.

Traceback (most recent call last):
File "ridi_find.py", line 5, in <module>
driver = webdriver.Chrome(chromedriver)
File "/home/ubuntu/play_python/venv/lib/python3.5/site-   packages/selenium/webdriver/chrome/webdriver.py", line 69, in __init__
desired_capabilities=desired_capabilities)
File "/home/ubuntu/play_python/venv/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 92, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/home/ubuntu/play_python/venv/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 179, in start_session
response = self.execute(Command.NEW_SESSION, capabilities)
File "/home/ubuntu/play_python/venv/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/home/ubuntu/play_python/venv/lib/python3.5/site-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
(Driver info: chromedriver=2.9.248304,platform=Linux 4.4.0-53-generic x86_64)

I installed chromedriver(linux64) and use python 3.5.2 in ec2 AWS server (ubuntu).

This is source code. But It's not work.

from selenium import webdriver
import pandas as pd

chromedriver = '/home/ubuntu/play_python/venv/bin/chromedriver'
driver = webdriver.Chrome(chromedriver)
driver.get('http://ridibooks.com/')

find_some = input("what do you want to know?")
find_some = find_some + '\n'

search = driver.find_element_by_css_selector("input[id='book_search_input']")
search.send_keys(find_some)
searches = driver.find_element_by_id('books_contents')

book_lists = []

for l in searches.find_elements_by_css_selector("span.title_text"):
    book_lists.append(l.text)

easy_index = pd.Series(0, index = range(1, len(book_lists) +1))
book_series = pd.Series(book_lists, index = easy_index.index)

print(book_series)
driver.quit()

Could anyone help me out to this problem?

python 3.5.2

Ubuntu 16.04.1 LTS

Chromedriver_linux64

Answer 1

I'm working on a similar issue, and it looks like you'll need a "fake" x environment like xvfb:

sudo yum install xorg-x11-server-Xvfbunzip
wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/2.10/chromedriver_linux64.zip && sudo unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/;

Secondly, it looks like some libraries aren't provided by AWS: https://forums.aws.amazon.com/message.jspa?messageID=713847

add the following to /etc/yum.repos.d/centos.repo

[CentOS-base]
name=CentOS-6 - Base
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=os
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
#released updates
[CentOS-updates]
name=CentOS-6 - Updates
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=updates
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
#additional packages that may be useful
[CentOS-extras]
name=CentOS-6 - Extras
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=x86_64&repo=extras
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

Next run:

sudo rpm --import http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6 

then finally:

sudo yum install GConf2

Edit:
My mistake, but the process is similar. Since you're using ubuntu, simply change where you're getting the packages:

sudo apt-get install python-pip
sudo apt-get install xvfb xserver-xephyr vnc4server
sudo pip install pyvirtualdisplay

My code looks something like this:

display = Display(visible=0, size=(1300, 1080))
display.start()
driver = webdriver.Chrome()
driver.set_window_size(1300, 1080)
driver.get("https://www.google.com")
...
driver.close()
driver.quit()
display.stop()