Reputation: 13
cannot extract some of the div tags with beautifulsoup
I want to obtain daily weather from the website 'https://www.timeanddate.com/weather' for the desired region and date. But I can not reach following div class with below code. What should I do?
I've tried to extract using BeautifulSoup. The information I want to extract is inside of the 'temp' and 'wdesc' div classes (which are the degrees and weather situations like; 'passing clouds' or so.) So I've tried following codes;
import requests
url = 'https://www.timeanddate.com/weather/spain/salou/historic?month=1&year=2014'
result = requests.get(url, verify = False)
soup = BeautifulSoup(result.text, "html.parser")
w1 = soup.findAll('div', attrs ={'class':'temp'})
w2 = soup.findAll('div', attrs ={'class':'wdesc'})
I expect to get the degrees of weather (13 / 11 °C) from the w1 and the situation of the weather from w2 (Scattered clouds.). But instead I get two empty lists from w1 and w2.
Upvotes: 0
Views: 451
Answers (3)
Reputation: 28565
If you just want the table (provided it's under <table> tags), it's much easier to use Pandas to pull that, than to use BeautifulSoup directly.
import pandas as pd
url = 'https://www.timeanddate.com/weather/spain/salou/historic?month=1&year=2014'
tables = pd.read_html(url)
df = tables[-1]
Output:
print (df.to_string())
Unnamed: 0_level_0 Conditions Comfort Unnamed: 7_level_0 Unnamed: 8_level_0
Time Unnamed: 1_level_1 Temp Weather Wind Unnamed: 5_level_1 Humidity Barometer Visibility
0 7:00 amWed, Jan 1 NaN 39 °F Clear. 3 mph ↑ 93% 30.07 "Hg 10 mi
1 7:30 am NaN 41 °F Clear. 3 mph ↑ 87% 30.07 "Hg 10 mi
2 8:00 am NaN 41 °F Passing clouds. 5 mph ↑ 87% 30.07 "Hg NaN
3 8:30 am NaN 43 °F Passing clouds. 6 mph ↑ 81% 30.07 "Hg NaN
4 9:00 am NaN 43 °F Passing clouds. 2 mph ↑ 87% 30.07 "Hg NaN
5 9:30 am NaN 46 °F Passing clouds. 5 mph ↑ 76% 30.07 "Hg NaN
6 10:00 am NaN 48 °F Passing clouds. 3 mph ↑ 76% 30.09 "Hg NaN
7 10:30 am NaN 54 °F Passing clouds. No wind ↑ 67% 30.09 "Hg NaN
8 11:00 am NaN 55 °F Passing clouds. No wind ↑ 63% 30.09 "Hg NaN
9 11:30 am NaN 55 °F Passing clouds. 3 mph ↑ 63% 30.09 "Hg NaN
10 12:00 pm NaN 57 °F Passing clouds. 6 mph ↑ 63% 30.07 "Hg NaN
11 12:30 pm NaN 57 °F Passing clouds. 8 mph ↑ 67% 30.07 "Hg NaN
12 1:00 pm NaN 59 °F Passing clouds. 9 mph ↑ 68% 30.04 "Hg NaN
13 2:00 pm NaN 59 °F Passing clouds. 10 mph ↑ 68% 30.01 "Hg NaN
14 2:30 pm NaN 59 °F Passing clouds. 9 mph ↑ 68% 30.01 "Hg NaN
15 3:00 pm NaN 57 °F Passing clouds. 7 mph ↑ 67% 30.01 "Hg NaN
16 3:30 pm NaN 57 °F Passing clouds. 6 mph ↑ 67% 29.98 "Hg NaN
17 4:00 pm NaN 57 °F Passing clouds. 3 mph ↑ 72% 30.01 "Hg NaN
18 4:30 pm NaN 55 °F Passing clouds. 3 mph ↑ 77% 30.01 "Hg NaN
19 5:00 pm NaN 55 °F Passing clouds. 1 mph ↑ 77% 29.98 "Hg NaN
20 5:30 pm NaN 54 °F Passing clouds. No wind ↑ 82% 30.01 "Hg NaN
21 6:00 pm NaN 52 °F Passing clouds. 1 mph ↑ 88% 29.98 "Hg NaN
22 6:30 pm NaN 52 °F Passing clouds. 1 mph ↑ 88% 29.98 "Hg NaN
23 7:30 pm NaN 50 °F Passing clouds. 3 mph ↑ 94% 29.98 "Hg NaN
24 8:00 pm NaN 50 °F Passing clouds. 3 mph ↑ 94% 29.98 "Hg NaN
25 8:30 pm NaN 52 °F Passing clouds. 7 mph ↑ 88% 29.98 "Hg NaN
26 9:00 pm NaN 52 °F Passing clouds. 5 mph ↑ 82% 29.98 "Hg NaN
27 9:30 pm NaN 50 °F Passing clouds. 5 mph ↑ 88% 29.98 "Hg NaN
28 10:00 pm NaN 48 °F Light rain. Passing clouds. 1 mph ↑ 94% 29.95 "Hg NaN
29
Additional:
To get multiple days, we'll get he data through the ajax. We'll just iterate through those requests. We also need to do a slight manipulation of the info returned as it's not exactly in a valid json format, but appears to be consistent, so shouldn't be an issue.
Note: you'll need to change the start_date and num_of_days to get what you want. this example starts on Jan 1, 2014, and will get that day, plus the next 9 days (so 10 days total)
import requests
from bs4 import BeautifulSoup
import json
import pandas as pd
import re
import datetime
start_date = '20140101'
num_of_days = 10
url = 'https://www.timeanddate.com/scripts/cityajax.php'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36'}
datetime_object = datetime.datetime.strptime(start_date, '%Y%m%d')
results = pd.DataFrame()
for x in range(num_of_days):
parse_time = datetime_object + datetime.timedelta(days=x)
str_time = parse_time.strftime('%Y%m%d')
month = parse_time.strftime('%#m')
year = parse_time.strftime('%Y')
payload = {
'n': 'spain/salou',
'mode': 'historic',
'hd': str_time,
'month': month,
'year': year,
'json': '1'}
jsonStr = requests.get(url, headers=headers, params=payload).text
jsonStr = jsonStr.replace('c:','"c":')
jsonStr = jsonStr.replace('h:','"h":')
jsonStr = jsonStr.replace('s:','"s":')
jsonData = json.loads(jsonStr)
for alpha in jsonData:
row = alpha['c']
try:
date = BeautifulSoup(row[0]['h'], 'html.parser').find('span').text
except:
pass
time = re.findall(r'\b((1[0-2]|0?[1-9]):([0-5][0-9]) ([AaPp][Mm]))', BeautifulSoup(row[0]['h'], 'html.parser').text)[0][0]
condition = BeautifulSoup(row[3]['h'], 'html.parser').text
temp = BeautifulSoup(row[2]['h'], 'html.parser').text.replace('\xa0', ' ')
wspd = BeautifulSoup(row[4]['h'], 'html.parser').text
wdir = BeautifulSoup(row[5]['h'], 'html.parser').text
wdesc = BeautifulSoup(row[5]['h'], 'html.parser').find('span')['title']
humd = BeautifulSoup(row[6]['h'], 'html.parser').text
barm = BeautifulSoup(row[7]['h'], 'html.parser').text
vis = BeautifulSoup(row[8]['h'], 'html.parser').text.replace('\xa0', ' ')
temp_df = pd.DataFrame([[date, time, temp, condition, wspd, wdir, wdesc, humd, barm, vis]], columns = ['Date', 'Time', 'Temp', 'Weather', 'Wind Speed', 'Wind Direction', 'Wind Description', 'Humidity', 'Barometer', 'Visibility'])
print ('Processed: %s %s' %(date, time))
results = results.append(temp_df).reset_index(drop=True)
Output:
print (results)
Date Time Temp ... Humidity Barometer Visibility
0 Wed, Jan 1 7:00 am 39 °F ... 93% 30.07 "Hg 10 mi
1 Wed, Jan 1 7:30 am 41 °F ... 87% 30.07 "Hg 10 mi
2 Wed, Jan 1 8:00 am 41 °F ... 87% 30.07 "Hg N/A
3 Wed, Jan 1 8:30 am 43 °F ... 81% 30.07 "Hg N/A
4 Wed, Jan 1 9:00 am 43 °F ... 87% 30.07 "Hg N/A
5 Wed, Jan 1 9:30 am 46 °F ... 76% 30.07 "Hg N/A
6 Wed, Jan 1 10:00 am 48 °F ... 76% 30.09 "Hg N/A
7 Wed, Jan 1 10:30 am 54 °F ... 67% 30.09 "Hg N/A
8 Wed, Jan 1 11:00 am 55 °F ... 63% 30.09 "Hg N/A
9 Wed, Jan 1 11:30 am 55 °F ... 63% 30.09 "Hg N/A
10 Wed, Jan 1 12:00 pm 57 °F ... 63% 30.07 "Hg N/A
11 Wed, Jan 1 12:30 pm 57 °F ... 67% 30.07 "Hg N/A
12 Wed, Jan 1 1:00 pm 59 °F ... 68% 30.04 "Hg N/A
13 Wed, Jan 1 2:00 pm 59 °F ... 68% 30.01 "Hg N/A
14 Wed, Jan 1 2:30 pm 59 °F ... 68% 30.01 "Hg N/A
15 Wed, Jan 1 3:00 pm 57 °F ... 67% 30.01 "Hg N/A
16 Wed, Jan 1 3:30 pm 57 °F ... 67% 29.98 "Hg N/A
17 Wed, Jan 1 4:00 pm 57 °F ... 72% 30.01 "Hg N/A
18 Wed, Jan 1 4:30 pm 55 °F ... 77% 30.01 "Hg N/A
19 Wed, Jan 1 5:00 pm 55 °F ... 77% 29.98 "Hg N/A
20 Wed, Jan 1 5:30 pm 54 °F ... 82% 30.01 "Hg N/A
21 Wed, Jan 1 6:00 pm 52 °F ... 88% 29.98 "Hg N/A
22 Wed, Jan 1 6:30 pm 52 °F ... 88% 29.98 "Hg N/A
23 Wed, Jan 1 7:30 pm 50 °F ... 94% 29.98 "Hg N/A
24 Wed, Jan 1 8:00 pm 50 °F ... 94% 29.98 "Hg N/A
25 Wed, Jan 1 8:30 pm 52 °F ... 88% 29.98 "Hg N/A
26 Wed, Jan 1 9:00 pm 52 °F ... 82% 29.98 "Hg N/A
27 Wed, Jan 1 9:30 pm 50 °F ... 88% 29.98 "Hg N/A
28 Wed, Jan 1 10:00 pm 48 °F ... 94% 29.95 "Hg N/A
29 Thu, Jan 2 7:00 am 43 °F ... 100% 29.89 "Hg N/A
.. ... ... ... ... ... ... ...
307 Sat, Jan 11 7:30 am 52 °F ... 82% 30.07 "Hg N/A
308 Sat, Jan 11 8:00 am 52 °F ... 82% 30.07 "Hg N/A
309 Sat, Jan 11 8:30 am 54 °F ... 82% 30.07 "Hg N/A
310 Sat, Jan 11 9:00 am 54 °F ... 77% 30.09 "Hg N/A
311 Sat, Jan 11 9:30 am 54 °F ... 82% 30.09 "Hg N/A
312 Sat, Jan 11 10:00 am 54 °F ... 82% 30.12 "Hg 4 mi
313 Sat, Jan 11 10:30 am 54 °F ... 82% 30.12 "Hg 4 mi
314 Sat, Jan 11 11:00 am 54 °F ... 82% 30.12 "Hg 4 mi
315 Sat, Jan 11 11:30 am 55 °F ... 77% 30.12 "Hg 4 mi
316 Sat, Jan 11 12:00 pm 57 °F ... 72% 30.12 "Hg 4 mi
317 Sat, Jan 11 12:30 pm 57 °F ... 72% 30.12 "Hg N/A
318 Sat, Jan 11 1:00 pm 57 °F ... 72% 30.09 "Hg N/A
319 Sat, Jan 11 1:30 pm 57 °F ... 72% 30.09 "Hg N/A
320 Sat, Jan 11 2:00 pm 57 °F ... 72% 30.09 "Hg N/A
321 Sat, Jan 11 2:30 pm 59 °F ... 72% 30.09 "Hg N/A
322 Sat, Jan 11 3:00 pm 59 °F ... 72% 30.07 "Hg N/A
323 Sat, Jan 11 3:30 pm 59 °F ... 72% 30.09 "Hg N/A
324 Sat, Jan 11 4:00 pm 57 °F ... 77% 30.09 "Hg N/A
325 Sat, Jan 11 4:30 pm 57 °F ... 77% 30.09 "Hg N/A
326 Sat, Jan 11 5:00 pm 55 °F ... 88% 30.09 "Hg N/A
327 Sat, Jan 11 5:30 pm 55 °F ... 88% 30.09 "Hg 6 mi
328 Sat, Jan 11 6:00 pm 55 °F ... 88% 30.12 "Hg 3 mi
329 Sat, Jan 11 6:30 pm 55 °F ... 94% 30.12 "Hg 3 mi
330 Sat, Jan 11 7:00 pm 55 °F ... 94% 30.12 "Hg 4 mi
331 Sat, Jan 11 7:30 pm 54 °F ... 100% 30.12 "Hg 4 mi
332 Sat, Jan 11 8:00 pm 54 °F ... 100% 30.15 "Hg 6 mi
333 Sat, Jan 11 8:30 pm 54 °F ... 100% 30.15 "Hg 6 mi
334 Sat, Jan 11 9:00 pm 54 °F ... 100% 30.15 "Hg 6 mi
335 Sat, Jan 11 9:30 pm 54 °F ... 94% 30.15 "Hg 6 mi
336 Sat, Jan 11 10:00 pm 54 °F ... 94% 30.15 "Hg 6 mi
[337 rows x 10 columns]
Upvotes: 0
Reputation: 4744
Your data is in script so one of the solution is using Selenium . If you dont install yet you can install it :
https://chromedriver.storage.googleapis.com/index.html?path=2.35/
This is the code :
from selenium import webdriver
driver_path = r'chromedriverpath'
browser = webdriver.Chrome(executable_path=driver_path)
browser.get("https://www.timeanddate.com/weather/spain/salou/historic?month=1&year=2014")
meta = browser.execute_script('return data')
my_json_string = meta['detail']
print my_json_string
OUTPUT :
[{u'hlsh': u'1 Oca', u'templow': 4, u'temp': 14, u'hum': 77, u'hls': u'1 Oca \xc7ar', u'ts': u'06:00', u'wd': 30, u'wind': 5, u'hl': True, u'date': 1388556000000, u'icon': 2, u'ds': u'1 Ocak 2014 \xc7ar\u015famba, 06:00 \u2014 12:00', u'baro': 1019, u'desc': u'Passing clouds.'}, {u'templow': 11, u'temp': 15, u'hum': 72, u'ts': u'12:00', u'wd': 210, u'wind': 9, u'date': 1388577600000, u'desc': u'Passing clouds.', u'ds': u'1 Ocak 2014 \xc7ar\u015famba, 12:00 \u2014 18:00', u'baro': 1016, u'icon': 2}, {u'templow': 9, u'temp': 11, u'hum': 90, u'ts': u'18:00', u'wd': 0, u'wind': 5, u'date': 1388599200000, u'desc': u'Passing clouds.', u'ds': u'1 Ocak 2014 \xc7ar\u015famba, 18:00 \u2014 00:00', u'baro': 1015, u'icon': 14}, {u'hlsh': u'2 Oca', u'wd': 0, u'hum': 0, u'hls': u'2 Oca Per', u'ts': u'00:00', u'wind': 0, u'hl': True, u'date': 1388620800000, u'icon': 36, u'ds': u'2 Ocak 2014 Per\u015fembe, 00:00 \u2014 06:00', u'baro': 0, u'desc': u'No weather data available'}, {u'templow': 6, u'temp': 15, u'hum': 93, u'ts': u'06:00', u'wd': 0, u'wind': 6, u'date': 1388642400000, u'desc': u'Passing clouds.', u'ds': u'2 Ocak 2014 Per\u015fembe, 06:00 \u2014 12:00', u'baro': 1013, u'icon': 2}, {u'templow': 15, u'temp': 18, u'hum': 61, u'ts': u'12:00', u'wd': 0, u'wind': 7, u'date': 1388664000000, u'desc': u'Passing clouds.', u'ds': u'2 Ocak 2014 Per\u015fembe, 12:00 \u2014 18:00', u'baro': 1013, u'icon': 2}, {u'templow': 13, u'temp': 15, u'hum': 80, u'ts': u'18:00', u'wd': 0, u'wind': 4, u'date': 1388685600000, u'desc': u'Passing clouds.', u'ds': u'2 Ocak 2014 Per\u015fembe, 18:00 \u2014 00:00', u'baro': 1014, u'icon': 14}, {u'hlsh': u'3 Oca', u'wd': 0, u'hum': 0, u'hls': u'3 Oca Cum', u'ts': u'00:00', u'wind': 0, u'hl': True, u'date': 1388707200000, u'icon': 36, u'ds': u'3 Ocak 2014 Cuma, 00:00 \u2014 06:00', u'baro': 0, u'desc': u'No weather data available'}, {u'templow': 9, u'temp': 18, u'hum': 76, u'ts': u'06:00', u'wd': 0, u'wind': 6, u'date': 1388728800000, u'desc': u'Passing clouds.', u'ds': u'3 Ocak 2014 Cuma, 06:00 \u2014 12:00', u'baro': 1015, u'icon': 2}, {u'templow': 17, u'temp': 20, u'hum': 55, u'ts': u'12:00', u'wd': 290, u'wind': 11, u'date': 1388750400000, u'desc': u'Passing clouds.', u'ds': u'3 Ocak 2014 Cuma, 12:00 \u2014 18:00', u'baro': 1016, u'icon': 2}, .. UP TO END
When you reach these list you can parse it with json or something else.Using selenium is one of the option
Upvotes: 1
Reputation: 719
I think you want to have a look at the table called wt-his instead. It seems to have rows of all the values your looking for.
Upvotes: 0
Related Questions
- Python BeautifulSoup cannot read data from div tag
- Beautiful Soup is not giving actual div tag
- Unable to grab div tag in Beautiful Soup in Python,
- "Extract text within div tag" but gives error
- beautifulsoup div extract
- python beautifulsoup no data retreived between div tags
- Unable to parse div tag using beautiful soup in python?
- Extract content of div including tags with BeautifulSoup 4
- Problems scraping specific content from a DIV - BeautifulSoup
- extracting data from div tags Python