CODEWITHSUNDEEP
pythonpython-3.xseleniumselenium-webdriver
Abhay Salvi
Abhay Salvi

Reputation: 1129

Why my selenium code isn't showing any text?

I want to scrape the whole website with selenium. I got one class of a product name in a website. I just want to get all the product names under one class name. Without manually copying any id's or XPATH's for each and every product. I have done it by doing this but:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

driver_exe = 'chromedriver'
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=r"C:\Users\intel\Downloads\Setups\chromedriver.exe", options=options)

driver.get("https://www.justdial.com/Bangalore/Bakeries")
x = driver.find_elements_by_class_name("store-name")

for i in x:
    print(i.text)

It's not displaying anything. Why??? Any parsers like beautiful soup will also be accepted mixed with selenium but I want selenium anyways...

Upvotes: 1

Views: 216

Answers (1)

Sers
Sers

Reputation: 12255

Bypass Access Denied in headless mode solution use different user-agent, reference. You can use your own user agent, google "my user agent" to get it.

user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) ' \
             'Chrome/80.0.3987.132 Safari/537.36'
driver_exe = 'chromedriver'
options = ChromeOptions()
options.add_argument("--headless")
options.add_argument(f'user-agent={user_agent}')
driver = webdriver.Chrome(options=options)

driver.get("https://www.justdial.com/Bangalore/Bakeries")
x = driver.find_elements_by_class_name("store-name")

for i in x:
    print(i.text)

Using and 

from bs4 import BeautifulSoup
import requests

user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) ' \
             'Chrome/80.0.3987.132 Safari/537.36'

response = requests.get('https://www.justdial.com/Bangalore/Bakeries', headers={'user-agent': user_agent})
soup = BeautifulSoup(response.text, 'lxml')
stores = soup.select('.store-name')
for store in stores:
    print(store.text.strip())

Output:

Big Mishra Pedha
Just Bake
The Cake Factory
Queen Of Cakeland
SREENIVASA BRAHMINS BAKERY ..
Aubree Eat Play Love Chocol..
Jain Bakes
Ammas Pastries
Facebake
Holige Mane Brahmins Bakery

Upvotes: 2

Related Questions