CODEWITHSUNDEEP
pythonjsonpython-3.xpandasfor-loop
Michael Mathews Jr.
Michael Mathews Jr.

Reputation: 329

Looping through url endpoints and converting json into pandas dataframe

Please see code below:

rr = pd.DataFrame()
for i in range(len(tt)):
    tes = pd.read_json(f'https://some/url/pageID={tt[i]}')
    data = rr.append(tes)
data

I start with an empty dataframe. I am trying to loop through a url that has many endpoints. In this case, 'tt' is a list of integers. Each integer points to an API endpoint with json. When I look at my dataframe called "data", it returns only the last row or the last endpoint in my loop. How can I fix this so I get a full dataframe?

Upvotes: 1

Views: 90

Answers (2)

Chris A
Chris A

Reputation: 91

The problem is that you're never actually updating your 'rr' Dataframe. If you change your new data variable to 'rr' it should work.

rr = pd.DataFrame()
for i in range(len(tt)):
    tes = pd.read_json(f'https://some/url/pageID={tt[i]}')
    rr = rr.append(tes)
rr

Upvotes: 2

bigbounty
bigbounty

Reputation: 17368

One way is to create individual dataframe for every request and append to a list. After all requests are done, you can concat all the individual dataframes to a single dataframe.

df_list = []
for i in range(len(tt)):
    tes = pd.read_json(f'https://some/url/pageID={tt[i]}')
    df_list.append(tes)
final_df = pd.concat(df_list)

Upvotes: 1

Related Questions