CODEWITHSUNDEEP
pythonseleniumselenium-webdriverweb-scrapingbeautifulsoup
Efe
Efe

Reputation: 29

Python, Selenium "::after" problem while scraping

I'm trying to scrape automobile information from a dynamic webpage. However, after running Selenium chrome browser, inspection elements are not shown as they are in original source page. Instead of html code of the car details (Informative area near the product image), " ::after " element is appeared in html source code.

You can see my scraping code below;


import requests
from requests import get
from bs4 import BeautifulSoup
from selenium import webdriver

driver_path = ("C:\\Desktop\\chromedriver.exe")
driver = webdriver.Chrome(driver_path)
driver.get('https://www.arabam.com/ilan/galeriden-satilik-citroen-c-elysee-1-6-hdi-attraction/fiat-onkol-oto-dan-c-elysee-1-6-attraction-92-hp-beyaz/14046287')
soup = BeautifulSoup(driver.page_source, 'html.parser')
table = soup.table

table_rows = table.find_all('li')
print(table_rows)

When i used given code to get relative information from the webpage, i could not see any html attributes which is necessary for further scraping loops.

What can be the reason of that problem and how can i solve that?

Thanks,

Edit;

HTML element content in selenium browser,

enter image description here

Normal Google Chrome HTML element content that i try to reach,

enter image description here

Upvotes: 1

Views: 299

Answers (1)

Aero Blue
Aero Blue

Reputation: 526

There is no table in the HTML page you provided, try using a different selector. You could try selecting by 

driver.find_elements_by_class_name("w100 semi-bold lh18")

This should give you an ordered list of the span elements

Upvotes: 0

Related Questions