CODEWITHSUNDEEP
pythonpandasweb-scraping
Jack the data lad
Jack the data lad

Reputation: 43

Scrape data from website to pandas dataframe

I'm trying to scrape weather data from the met office website, but I keep getting errors.

Here is what I have tried so far

stats = ['Tmax', 'Tmin', 'Rainfall']
regions = ['England', 'Wales', 'Scotland']
base_url = r'https://www.metoffice.gov.uk/pub/data/weather/uk/climate/datasets/{}/date/{}.txt'
dframes = []

for r in regions:
    for s in stats:
        url = base_url.format(s,r)
        df = pd.read_table(requests.get(url).content)
        dframes.append(df)

I'm gettin an error:

"OSError: Expected file path name or file-like object, got <class 'bytes'> type"

Upvotes: 1

Views: 425

Answers (1)

eyllanesc
eyllanesc

Reputation: 243955

You have to use that bytes through a stream for example io.BytesIO:

import pandas as pd
import requests
import io

stats = ['Tmax', 'Tmin', 'Rainfall']
regions = ['England', 'Wales', 'Scotland']
base_url = r'https://www.metoffice.gov.uk/pub/data/weather/uk/climate/datasets/{}/date/{}.txt'
dframes = []

for r in regions:
    for s in stats:
        url = base_url.format(s,r)
        df = pd.read_table(io.BytesIO(requests.get(url).content))
        dframes.append(df)
print(dframes)

Upvotes: 6

Related Questions