CODEWITHSUNDEEP
stringpython-3.xclassbeautifulsoup
Sibren De Preter
Sibren De Preter

Reputation: 129

BeautifulSoup Div Class returns empty

I checked similar questions, but could not find a solution...

I'm trying to scrape the minutes of extra travel time (46) from the following page: https://www.tomtom.com/en_gb/trafficindex/city/istanbul

I've tried by 2 methods (Xpath & find class), but both give an empty return.

import requests
from bs4 import BeautifulSoup
from lxml.html import fromstring

page = requests.get("https://www.tomtom.com/en_gb/trafficindex/city/istanbul")
tree = fromstring(page.content)

soup = BeautifulSoup(page.content, 'html.parser')



#print([type(item) for item in list(soup.children)])

html = list(soup.children)[2]

g_data = soup.find_all("div", {"class_": "big.ng-binding"})

congestion = tree.xpath("/html/body/div/div[2]/div[2]/div[2]/section[2]/div/div[2]/div/div[2]/div/div[2]/div[1]/div[1]/text()")
print(congestion)
print(len(g_data))

Am I missing something obvious?

Many thanks for helping out!

Upvotes: 2

Views: 3673

Answers (1)

Michael Dz
Michael Dz

Reputation: 3834

Unfortunately BeautifulSoup alone is not enough to accomplish it. The website uses JavaScript to generate content so you will have to use additional tools like for example Selenium.

import bs4 as bs
import re
from selenium import webdriver

url = 'https://www.tomtom.com/en_gb/trafficindex/city/istanbul'

driver = webdriver.Firefox()
driver.get(url)           
html = driver.page_source
soup = bs.BeautifulSoup(html)

I can see two approaches to extract extra time:

1.Looking for div with class="text-big ng-binding".

div = soup.find_all('div', attrs={'class' : 'text-big ng-binding'})
result = div[0].text

2.Finding div containing Per day text first and then going two divs up

div = soup.find_all(text=re.compile('Per day'))
result = div.find_previous('div').find_previous('div').text

Upvotes: 5

Related Questions