CODEWITHSUNDEEP
pythonselenium-webdriverbeautifulsouppython-requestsurllib
user2293224
user2293224

Reputation: 2220

Selenium Webdriver exception while retrieving google play reviews

I am trying to retrieve app related information from Google Play store using selenium and BeautifulSoup. When I try to retrieve the information, I got webdriver exception error. I checked the chrome version and chrome driver version (both are compatible). Here is the weblink that is causing the issue, code to retrieve information, and error thrown by the code:

Link: https://play.google.com/store/apps/details?id=com.tudasoft.android.BeMakeup&hl=en&showAllReviews=true

Code:

driver = webdriver.Chrome('path')
driver.get('https://play.google.com/store/apps/details?id=com.tudasoft.android.BeMakeup&hl=en&showAllReviews=true')

soup = bs.BeautifulSoup(driver.page_source, 'html.parser')

I am getting error on third line. Here is the parts of the error message:

Start of the error message:

---------------------------------------------------------------------------
WebDriverException                        Traceback (most recent call last)
<ipython-input-280-4e8a1ef443f2> in <module>()
----> 1 soup = bs.BeautifulSoup(driver.page_source, 'html.parser')

~/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py in page_source(self)
    676             driver.page_source
    677         """
--> 678         return self.execute(Command.GET_PAGE_SOURCE)['value']
    679 
    680     def close(self):

~/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py in execute(self, driver_command, params)
    318         response = self.command_executor.execute(driver_command, params)
    319         if response:
--> 320             self.error_handler.check_response(response)
    321             response['value'] = self._unwrap_value(
    322                 response.get('value', None))

~/anaconda3/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py in check_response(self, response)
    240                 alert_text = value['alert'].get('text')
    241             raise exception_class(message, screen, stacktrace, alert_text)
--> 242         raise exception_class(message, screen, stacktrace)
    243 
    244     def _value_or_default(self, obj, key, default):

WebDriverException: Message: unknown error: bad inspector message:

End of the error message:

(Session info: chrome=79.0.3945.117)

Could anyone guide me how to fix the issue?

Upvotes: 0

Views: 270

Answers (3)

undetected Selenium
undetected Selenium

Reputation: 193308

You can use with as follows:

  • Code Block:

    # -*- coding: UTF-8 -*
from bs4 import BeautifulSoup
from urllib.request import urlopen as uReq

url = "https://play.google.com/store/apps/details?id=com.tudasoft.android.BeMakeup&hl=en&showAllReviews=true"
uClient = uReq(url)
page_html = uClient.read()
uClient.close()
page_soup = BeautifulSoup(page_html, "html.parser")
print(page_soup)

  • Console Output:

    <!DOCTYPE doctype html>
<html dir="ltr" lang="en"><head><base href="https://play.google.com/"/><meta content="origin" name="referrer"/><link href="/opensearch.xml" rel="search" title="Google Play" type="application/opensearchdescription+xml"/>
.
.
.
<style nonce="96JYwPKBYhVDb+ABipwCww">@font-face{font-family:'Roboto';font-style:normal;font-weight:100;src:local('Roboto Thin'),local('Roboto-Thin'),url(//fonts.gstatic.com/s/roboto/v18/KFOkCnqEu92Fr1MmgVxIIzc.ttf)format('truetype');}@font-face{font-family:'Roboto';font-style:normal;font-weight:300;src:local('Roboto Light'),local('Roboto-Light'),url(//fonts.gstatic.com/s/roboto/v18/KFOlCnqEu92Fr1MmSU5fBBc9.ttf)format('truetype');}@font-face{font-family:'Roboto';font-style:normal;font-weight:400;src:local('Roboto Regular'),local('Roboto-Regular'),url(//fonts.gstatic.com/s/roboto/v18/KFOmCnqEu92Fr1Mu4mxP.ttf)format('truetype');}@font-face{font-family:'Roboto';font-style:normal;font-weight:500;src:local('Roboto Medium'),local('Roboto-Medium'),url(//fonts.gstatic.com/s/roboto/v18/KFOlCnqEu92Fr1MmEU9fBBc9.ttf)format('truetype');}@font-face{font-family:'Roboto';font-style:normal;font-weight:700;src:local('Roboto Bold'),local('Roboto-Bold'),url(//fonts.gstatic.com/s/roboto/v18/KFOlCnqEu92Fr1MmWUlfBBc9.ttf)format('truetype');}@font-face{font-family:'Material Icons Extended';font-style:normal;font-weight:400;src:url(//fonts.gstatic.com/s/materialiconsextended/v50/kJEjBvgX7BgnkSrUwT8UnLVc38YydejYY-oE_LvM.ttf)format('truetype');}.material-icons-extended{font-family:'Material Icons Extended';font-weight:normal;font-style:normal;font-size:24px;line-height:1;letter-spacing:normal;text-transform:none;display:inline-block;white-space:nowrap;word-wrap:normal;direction:ltr;}@font-face{font-family:'Product Sans';font-style:normal;font-weight:400;src:local('Product Sans'),local('ProductSans-Regular'),url(//fonts.gstatic.com/s/productsans/v9/pxiDypQkot1TnFhsFMOfGShVF9eL.ttf)format('truetype');}</style><script nonce="96JYwPKBYhVDb+ABipwCww">(function(){/*

 Copyright The Closure Library Authors.
 SPDX-License-Identifier: Apache-2.0
*/

Upvotes: 0

user11527247
user11527247

Reputation: 238

I think this is due to the chromedriver encoding problem. See https://bugs.chromium.org/p/chromium/issues/detail?id=723592#c9 for additional information about this bug.

Instead of selenium you can get page source using BeautifulSoup as follows.

import requests
from bs4 import BeautifulSoup

r = requests.get('https://play.google.com/store/apps/details?id=com.tudasoft.android.BeMakeup&hl=en&showAllReviews=true')
soup = BeautifulSoup(r.content, "lxml")
print(soup)

Upvotes: 2

Manali Kagathara
Manali Kagathara

Reputation: 761

try this

 driver = webdriver.Chrome('path')
 driver.get('https://play.google.com/store/apps/details?id=com.tudasoft.android.BeMakeup&hl=en&showAllReviews=true')

 # retrieve data you want, for example
 review_user_list = driver.find_elements_by_class_name("X43Kjb")

Upvotes: 1

Related Questions