CODEWITHSUNDEEP
pythonapiweb-scraping
Yel
Yel

Reputation: 55

Web Scraping from an API Loop

I'm scraping from the World Bank for a paper and I'm trying to make a loop of the web scraping of different indicators but I can't seem to make it work until a certain part of the code. Hope someone can help please?

#Single Code for each indicator
indcator = 'SP.POP.TOTL?date=2000:2020'
url = "http://api.worldbank.org/v2/countries/all/indicators/%s&format=json&per_page=5000" % indicator
response = requests.get(url)
print(response)
result = response.content
result = json.loads(result)
pop_total_df = pd.DataFrame.from_dict(result[1])

This is the loop i'm trying to build but I got an error in the last part of below code:

#indicator list
indicator =  {'FP.CPI.TOTL.ZG?date=2000:2020','SP.POP.TOTL?date=2000:2020'}

#list of urls with the indicators
url_list = []
   for i in indicator:
   url = "http://api.worldbank.org/v2/countries/all/indicators/%s&format=json&per_page=5000" % i
   url_list.append(url)


result_list = []
for i in url_list:
   response = requests.get(i)
   print(response)
   result_list.append(response.content)

#Erroneous code
result_json = []
for i in range(3):
   result_json.append(json.loads(result_list[i])))

Upvotes: 0

Views: 103

Answers (1)

sverdejot
sverdejot

Reputation: 83

As you are making 2 requests (FP.CPI.TOTL.ZG?date=2000:2020 and SP.POP.TOTL?date=2000:2020) your result_list length is 2, so its index are 0 and 1. Use range(2) or range(len(result_list)) instead:

import requests, json

#indicator list
indicator =  {'FP.CPI.TOTL.ZG?date=2000:2020','SP.POP.TOTL?date=2000:2020'}

#list of urls with the indicators
url_list = []
for i in indicator:
   url = "http://api.worldbank.org/v2/countries/all/indicators/%s&format=json&per_page=5000" % i
   url_list.append(url)


result_list = []
for i in url_list:
   response = requests.get(i)
   print(response)
   result_list.append(response.content)

#Erroneous code
result_json = []
for i in range(len(result_list)):
   result_json.append(json.loads(result_list[i]))

Upvotes: 3

Related Questions