CODEWITHSUNDEEP
python-3.xweb-scrapingbeautifulsoup
Sibren De Preter
Sibren De Preter

Reputation: 129

Scrape data from webpage under 'popup' box

I'm trying to scrape data from a website. The problem is the data is only visible when the mouse pointer is hovering over it... On the following page, I would like to extract the historical congestion levels (right bottom, when mouse on e.g. 2011) https://www.tomtom.com/en_gb/trafficindex/city/mexico-city

I'm somewhat familiar with beautiful soup. Any ideas on how to tackle this, if possible after all...

Many thanks and sorry for the high level question, but I wanted to check feasibility before diving into it.

Upvotes: 0

Views: 1258

Answers (1)

t.m.adam
t.m.adam

Reputation: 15376

I think the best approach here is to request the json file (/en_gb/trafficindex/data.json) directly.
The file contains a list of 390 items, one for each city. You could create a dictionary from this list with 'cityCode' as keys and 'congestionHistory' as values, and access the data by city code.

An example with requests: 

import requests

url = "https://www.tomtom.com/en_gb/trafficindex/data.json"
r = requests.get(url)
data = r.json()
congestion_data = {
    i['cityTraffic']['cityCode']: i['cityTraffic']['congestionHistory'] 
    for i in data
}

print(congestion_data['MEX'])

[{'year': 2010, 'congestion': 57}, {'year': 2011, 'congestion': 59}, ...

And saving it as a csv file: 

import csv

with open('my_file.csv', 'w', newline='') as f: 
    w = csv.writer(f)
    w.writerow(['city_code', 'congestion_history'])
    for k,v in congestion_data.items():
        w.writerow((k, ', '.join('{1}:{0}'.format(*i.values()) for i in v)))

Upvotes: 1

Related Questions