CODEWITHSUNDEEP
pythonbeautifulsouppython-requests
bbartling
bbartling

Reputation: 3502

Python web scrape numerical weather data

I am attempting to print the int value of current outside air temperature. (55)

Any chance for a tip on what I am doing wrong? (sorry not a lot of wisdom here!)

import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
import datetime as dt

#this is used at the end with plotting results to current hour
h = dt.datetime.now().hour



r = requests.get(
'https://www.google.com/search?q=weather+duluth')
soup = BeautifulSoup(r.text, 'html.parser')

stuff = []

for item in soup.select('vk_bk sol-tmp'):
    item = int(item.contents[1].get_text(strip=True)[:-1])
    #print(item)#this is weather data
    stuff.append(item)

This is the web URL for weather and the current outdoor temperature is tied to the div class highlighted below.

If I attempt to print stuff I just get an empty list returned. enter image description here

Upvotes: 0

Views: 117

Answers (1)

sushanth
sushanth

Reputation: 8302

Adding User-Agent header should give expected result

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

r = requests.get('https://www.google.com/search?q=weather%20duluth', headers=headers)
soup = BeautifulSoup(r.text, 'html.parser')

soup.find("span", {"class": "wob_t"}).text

Upvotes: 2

Related Questions