CODEWITHSUNDEEP
pythonpandasdataframepython-requests
dubbbdan
dubbbdan

Reputation: 2740

Decoded text file to pandas data frame

I am getting a tab separated data file using requests and I want to convert it to a pandas data frame. However, I can't seem to figure out how to convert the decoded data file into a pandas data frame object. 

import requests
import pandas as pd 
from datetime import date, timedelta


def build_url(site,yesterday):
    url = 'https://waterdata.usgs.gov/az/nwis/dv?cb_00060=on&format=rdb&site_no=' + gc + '&referred_module=sw&period=&begin_date=1989-01-01&end_date=' + yesterday 
    return url

yesterday = date.today() - timedelta(1)

yesterday=yesterday.strftime('%Y-%m-%d')

url = build_url(site,yesterday)
t = requests.get(url)
decoded = t.content.decode('utf-8')
tmp_df = pd.read_csv(decoded,sep='\t',encoding = 'utf8')

My understanding is that decoded is a text file living in memory, but when I pass it to pd.read_csv with the specified delimiter it begins to print out the data frame and ends with:

USGS    09402500    2017-07-19  15200   P
USGS    09402500    2017-07-20  15200   P
USGS    09402500    2017-07-21  15100   P
USGS    09402500    2017-07-22  15000   P
USGS    09402500    2017-07-23  14100   P
USGS    09402500    2017-07-24  13700   P
 does not exist

How can I get pandas to convert decoded into a dataframe?

Upvotes: 0

Views: 763

Answers (1)

user1620443
user1620443

Reputation: 794

read_csv wants a filename or a buffer. You can either save decoded to a file, or use a StringIO object:

import StringIO    
tmp_df = pd.read_csv(StringIO.StringIO(decoded), sep='\t')

Upvotes: 2

Related Questions