CODEWITHSUNDEEP
pythonapiloopsiteration
M.Ai
M.Ai

Reputation: 45

python loop to pull API data for iterating URLs

I'm trying to set up a for loop to pull in elected representatives' data for about 600,000 postal codes. The base URL stays the same, and the only part that changes is the postal code.

I'd ideally like to create a list of all the postal codes, and then use requests.get to pull in data for all the postal codes in my list. I came up with this code below, but it's only pulling in data for the last postal code in my list. I'm not really sure why this is happening and am a python beginner - so any help would be appreciated! 

#loop test 
postcodes = ['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']

for i in range(len(postcodes)):
   rr = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(postcodes[i]))

data1=json.loads(rr.text)

data1

Upvotes: 3

Views: 20687

Answers (3)

P.hunter
P.hunter

Reputation: 1365

you are just viewing the last response. 

#loop test 
postcodes = ['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']

api_data = dict()

for i in postcodes:
   rr = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(i))

   data = json.loads(rr.text)
   api_data.update({i: data})   
   # or print(data)

print(api_data)

here I've added all the response to a dict, with key as postal code and value as the response.

Upvotes: 1

Maciej M
Maciej M

Reputation: 786

Your code doesnt works because it overwrites data1.

Try this:

#loop test 
responses = list() # stores responses for postal codes
postcodes = ['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']

for postcode in postcodes:
   rr = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(postcode))
   data=json.loads(rr.text)
   responses.append(data)

Your responses now are saved in responses list.

Tips:
You can iterate over list without using index.

Upvotes: 5

Josef Korbel
Josef Korbel

Reputation: 1208

You are overwriting data1 variable every iteration, that is why you end up with only the last one, you need to store it differently.

Example:

postcodes =['P0L1B0','P5A3P1', 'P5A3P2', 'P5A3P3']
results = []

for postcode in postcodes:
    res = requests.get('https://represent.opennorth.ca/postcodes/{}'.format(postcode))

    if res.status_code == 200:
        results.append(res.json())
    else: 
        print("Request to {} failed".format(postcode))

Upvotes: 1

Related Questions