CODEWITHSUNDEEP
python-3.xweb-scrapingbeautifulsoup
Surabhi Choudhary
Surabhi Choudhary

Reputation: 548

Python BeautifulSoup - Table return none when scraping by id

I want to scrape Daily Observation table from below given url https://www.wunderground.com/history/daily/in/chennai/VOMM/date/2017-1-1

I want to use table id for scraping. I am using this code

from bs4 import BeautifulSoup
import requests
import lxml

url = 'https://www.wunderground.com/history/daily/in/chennai/VOMM/date/2017-1-1';
content = requests.get(url).content
soup = BeautifulSoup(content, 'lxml')
table = soup.find('table', {'id' : 'history-observation-table'})
print(table)

But this is returning None. How can I scrape table?

Upvotes: 0

Views: 418

Answers (1)

ewwink
ewwink

Reputation: 19154

It dynamic page, you can use json data from URL like

https://api.weather.com/v1/geocode/12.99361134/80.17694092/observations/historical.json?apiKey=*********&startDate=20170101&endDate=20170101&units=e

you can see real API key it in browser Console -> Network

or use selenium

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait 

driver = webdriver.Chrome()
driver.get("https://www.wunderground.com/history/daily/in/chennai/VOMM/date/2017-1-1")

table = WebDriverWait(driver, 15).until(lambda d: d.find_element_by_id('history-observation-table'))
print(table.text)

Upvotes: 1

Related Questions