CODEWITHSUNDEEP
pythonweb-scraping
Jacob Sedore
Jacob Sedore

Reputation: 37

Extracting <span> element text

I have been able to successfully get the tag that I want. However when I am looking at the websites code via inspect on my browser, I can see there is a data value. When I scrape the websites HTML code in, the data value isn't there.

import requests
from bs4 import BeautifulSoup

page = requests.get("https://www.theweathernetwork.com/ca/weather/ontario/ottawa")

if page.status_code == 200:
    print("Page downloaded")
else:
    print("Page download failed")

soup = BeautifulSoup(page.content, 'html.parser')

# Locate current temperature
test = soup.find("span", {"class":"temp"})

print(test)

The result set that I am getting back is <span class="temp"></span> Am I using the wrong function to return my results back? I have also tried using the soup.select() method.

Upvotes: 1

Views: 58

Answers (1)

baduker
baduker

Reputation: 20052

You don't get anything back, because the values are dynamically rendered by JS. However, you can make a request to their API to get all the data.

import requests

data = requests.get("https://www.theweathernetwork.com/api/data/caon0512/cm/ci?ts=1652").json()
print(data['obs']['tc'])

This gets you current observable temperature in Ottawa in C: 11.

To get more data:

import requests

data = requests.get("https://www.theweathernetwork.com/api/data/caon0512/cm/ci?ts=1652").json()

print(data['obs']['updatetime'])
sunrise = data['obs']['sunrise_time']
temp_c = data['obs']['tc']
wind, wind_unit = data['obs']['w'], data['obs']['wu']
humidity = data['obs']['h']
visibility, visibility_unit = data['obs']['v'], data['obs']['vu']
air_quality = data['obs']['aq_index']

print(f"Sunrise: {sunrise} | Current temp: {temp_c} C | Wind: {wind} {wind_unit}")
print(f"Humidity: {humidity}% | Visibility: {visibility} {visibility_unit}")
print(f"Air quality index: {air_quality}")

This gets you:

Sun Oct 18 1:15 PM
Sunrise: 7:23 AM | Current temp: 13 C | Wind: 17 km/h
Humidity: 47% | Visibility: 24 km
Air quality index: 2

Upvotes: 3

Related Questions