CODEWITHSUNDEEP
python-3.xweb-scrapingaws-lambdaselenium-chromedriver
Luv Tomar
Luv Tomar

Reputation: 127

AWS Lambda Python 3.7 Web Scraping - "Could not get version for Chrome with this command: google-chrome --version"

Via an S3 bucket, I've uploaded a lambda function along with its dependencies as a ZIP file. The lambda function is a web scraper with the following initial code to get the scraper started:

import json
import os
import pymysql
import boto3
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common.exceptions import StaleElementReferenceException
from selenium.webdriver.support import expected_conditions as EC

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--window-size=1280x1696')
chrome_options.add_argument('--user-data-dir=/tmp/user-data')
chrome_options.add_argument('--hide-scrollbars')
chrome_options.add_argument('--enable-logging')
chrome_options.add_argument('--log-level=0')
chrome_options.add_argument('--v=99')
chrome_options.add_argument('--single-process')
chrome_options.add_argument('--data-path=/tmp/data-path')
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--homedir=/tmp')
chrome_options.add_argument('--disk-cache-dir=/tmp/cache-dir')
chrome_options.binary_location = os.getcwd() + "/bin/headless-chromium"
browser = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=chrome_options)

When I try to test the lambda function, I get the following error in the console:

{
  "errorMessage": "Could not get version for Chrome with this command: google-chrome --version",
  "errorType": "ValueError",
  "stackTrace": [
    "  File \"/var/task/lambda_function.py\", line 67, in lambda_handler\n    browser = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=chrome_options)\n",
    "  File \"/var/task/webdriver_manager/chrome.py\", line 24, in install\n    driver_path = self.download_driver(self.driver)\n",
    "  File \"/var/task/webdriver_manager/manager.py\", line 32, in download_driver\n    driver_version, is_latest = self.__get_version_to_download(driver)\n",
    "  File \"/var/task/webdriver_manager/manager.py\", line 23, in __get_version_to_download\n    return self.__get_latest_driver_version(driver), True\n",
    "  File \"/var/task/webdriver_manager/manager.py\", line 17, in __get_latest_driver_version\n    return driver.get_latest_release_version()\n",
    "  File \"/var/task/webdriver_manager/driver.py\", line 54, in get_latest_release_version\n    self._latest_release_url + '_' + chrome_version())\n",
    "  File \"/var/task/webdriver_manager/utils.py\", line 98, in chrome_version\n    .format(cmd)\n"
  ]
}

In response, I tried editing the utils.py file in the webdriver_manager dependency folder, by using other commands like 'chrome --version' and 'chromium-browser --version' instead of 'google-chrome --version' under the function definition of 'chrome_version()', but got the similar error of not being able to the get the chrome version from the new command:

def chrome_version():
    pattern = r'\d+\.\d+\.\d+'
    cmd_mapping = {
        OSType.LINUX: 'google-chrome --version',
        OSType.MAC: r'/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version',
        OSType.WIN: r'reg query "HKEY_CURRENT_USER\Software\Google\Chrome\BLBeacon" /v version'
    }

    cmd = cmd_mapping[os_name()]
    stdout = os.popen(cmd).read()
    version = re.search(pattern, stdout)
    if not version:
        raise ValueError(
            'Could not get version for Chrome with this command: {}'
            .format(cmd)
        )
    return version.group(0)

Can anyone tell me what command I should be using instead of 'google-chrome --version'?

Upvotes: 3

Views: 3201

Answers (1)

Allan Chua
Allan Chua

Reputation: 10175

By default, Google Chrome does not exists on the container that runs our lambda functions.

I'm implementing similar solutions but with JavaScript and the way I solve is by using a micro-browser (Chromium) using the following packages:

    "chrome-aws-lambda": "^1.19.0",
    "puppeteer-core": "^1.19.0"

For Python, here is a tutorial that might help in your situation.

https://robertorocha.info/setting-up-a-selenium-web-scraper-on-aws-lambda-with-python/

Upvotes: 1

Related Questions