Web scraping with Selenium + Python

Question

The objective is to scrape the historical weather from http://www.weather.gov.sg/climate-historical-daily/

To obtain the data for the particular month, first have to select the cityname, month and year

There are 63 cities,12 months and 41 years

city = [el.text for el in driver.find_elements_by_xpath("/html/body/div/div/div[3]/div[1]/div[1]/div/div/ul/li/a")]
len(city)
Out[182]: 63

month = [el.text for el in driver.find_elements_by_xpath('//*[@id="monthDiv"]/ul/li')]
year = [el.text for el in driver.find_elements_by_xpath('//*[@id="yearDiv"]/ul/li')]

click display button

button = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "display")))
button.click()

How to select option from these bootstrap drop downlist and copy the weather data in


                
                
                
                
                
                
                
                
                
                
              Date Daily Rainfall Total (mm) Highest  30-min Rainfall (mm) Highest  60-min Rainfall (mm) Highest 120-min Rainfall (mm) Mean Temperature (°C) Maximum Temperature (°C) Minimum Temperature (°C) Mean Wind Speed (km/h) Max Wind Speed (km/h)
1 Aug 0.0 0.0 0.0 0.0 28.5 30.4 26.0 12.3 40.7
2 Aug 0.0 0.0 0.0 0.0 28.9 31.7 26.9 10.3 31.5
3 Aug 0.0 0.0 0.0 0.0 29.2 31.7 27.2 12.0 31.5
4 Aug 4.8 4.6 4.8 4.8 27.9 30.2 24.1 8.8 44.4
5 Aug 0.0 0.0 0.0 0.0 28.8 31.8 26.7 8.6 25.9
6 Aug 0.0 0.0 0.0 0.0 29.2 31.4 27.6 8.1 27.8
7 Aug 0.0 0.0 0.0 0.0 29.4 32.7 27.3 11.4 29.6
8 Aug 0.0 0.0 0.0 0.0 29.7 32.9 27.6 11.0 27.8
9 Aug 0.0 0.0 0.0 0.0 29.6 32.8 27.7 12.3 31.5
10 Aug 0.0 0.0 0.0 0.0 29.7 33.0 27.8 12.9 33.3
11 Aug 0.0 0.0 0.0 0.0 29.5 32.7 28.2 11.0 31.5
12 Aug 0.0 0.0 0.0 0.0 27.9 30.0 26.8 8.7 31.5
13 Aug 34.6 22.2 30.8 33.4 28.3 32.2 22.5 6.4 40.7
14 Aug 13.8 7.2 12.2 12.6 25.9 28.5 23.4 5.1 35.2
15 Aug 0.0 0.0 0.0 0.0 28.0 31.5 24.6 6.5 25.9
16 Aug 0.0 0.0 0.0 0.0 28.0 30.0 26.4 8.0 27.8
17 Aug 5.2 4.0 4.6 4.6 27.4 31.4 24.3 6.2 29.6
18 Aug 2.0 1.0 1.0 2.0 27.1 30.1 25.3 6.4 48.2
19 Aug 1.8 1.4 1.6 1.8 28.0 31.3 25.4 5.7 25.9
20 Aug 2.2 2.0 2.0 2.0 28.1 31.9 25.5 10.6 37.0
21 Aug 0.0 0.0 0.0 0.0 29.6 33.0 27.7 15.2 31.5
22 Aug 2.0 1.4 1.6 1.6 27.9 32.1 25.3 9.3 38.9
23 Aug 24.4 8.2 11.2 15.2 25.6 27.0 23.0 5.1 48.2
24 Aug 0.0 0.2 0.2 0.2 28.1 32.4 24.5 9.0 33.3
25 Aug 0.0 0.0 0.0 0.0 27.9 31.9 25.7 8.6 44.4
26 Aug 4.6 4.4 4.6 4.6 27.0 31.3 24.0 9.6 51.9
27 Aug 1.4 1.4 1.4 1.4 27.8 30.4 25.6 8.4 27.8
28 Aug 0.0 0.0 0.0 0.0 28.9 32.3 26.2 9.6 33.3
29 Aug 6.6 2.8 3.4 4.8 27.2 30.8 25.1 8.0 -
30 Aug 0.0 0.0 0.0 0.0 28.6 32.1 26.4 11.2 35.2
31 Aug 0.0 0.0 0.0 0.0 29.0 32.2 27.2 11.7 29.6

baduker · Accepted Answer

Here's a different approach.

Why not get all the .csv files for all the cities and all the dates? The link to the file is static and uses the code of the city that's in the drop-down menu. You can parse this, grab the code, put it in the url and get the .csv file. Oh, and you have to loop over all the years too.

By the way, not all cities have data for the past 40 years.

import re
import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "PostmanRuntime/7.26.5",
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
}

response = requests.get("http://www.weather.gov.sg/climate-historical-daily/")

soup = BeautifulSoup(response.text, "html.parser").find("ul", {"class": "dropdown-menu long-dropdown"}).find_all("li")
cities_and_codes = {
    t.find("a").getText(strip=True): re.search(r'(S\d+)', t.find("a")['onclick']).group(1)
    for t in soup
}


def get_dates():
    yield from (
        [(y, f"0{m}" if m < 10 else m) for y in range(1980, 2021) for m in range(1, 13)]
    )


files_url = "http://www.weather.gov.sg/files/dailydata/DAILYDATA_"
for city, code in cities_and_codes.items():
    for date in get_dates():
        year, month = date
        csv_url = f"{files_url}{code}_{year}{month}.csv"
        response = requests.get(csv_url)
        if response.status_code == 200:
            print(f"Fetching data for {city} for {month}/{year}")
            print(f"Found data. Fetching {csv_url}")
            with open(f"{city.replace(' ', '_')}_{csv_url.split('/')[-1]}", "wb") as f:
                f.write(response.content)
        else:
            print(f"No data available for {city} for {month}/{year}...")
            continue

You can play around with this and just get the files for those cities you want, or all of them, but that might take a while.

Web scraping with Selenium + Python

Answers (2)

Related Questions