CODEWITHSUNDEEP
pythonrestpagination
d3wannabe
d3wannabe

Reputation: 1317

Restful paginate with python

I'm getting data using Python via a rest API call as follows...

result = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc"))

There's a 100 max results built into the API so I have to paginate in python.

After making the first call, result.nextPage returns a code that I then have to pass to the next API call like...

result2 = json.load(urllib2.urlopen("https://api.somesite.com/v0/someLink?api_key=aaabbbccc&nextPage=someCode"))

And so on until I'm through all pages.

Does python have any in-built mechanism to handle pagination like this?

I could write a clumsy loop myself but am unsure about

  1. How to handle waiting for each page so I know I can proceed with the next call
  2. How to handle the 'no more pages' event without erroring
  3. How to combine the results of all calls into 1 variable after making all calls

Appreciate any thoughts at all on the most elegant way to go about this.

Upvotes: 3

Views: 2544

Answers (1)

Tomalak
Tomalak

Reputation: 338118

Does python have any in-built mechanism to handle pagination like this?

No.

Write a while True: loop with a break statement if there are no more pages to load.

How to handle waiting for each page so I know I can proceed with the next call

urllib2.urlopen is not asynchronous. Your code will block (i.e. wait) until the request is done.

How to handle the 'no more pages' event without erroring

That depends on the API you are using. I would expect that result.nextPage is empty/not set on the last page.

How to combine the results of all calls into 1 variable after making all calls

Append them to a list as you go. Pseudo code:

url = "initial URL"
results = []

while True:
    current_page = request(url)
    results.append(current_page)
    if (another page available):
        url = "new url with next page code"
    else:
        break

I wholeheartedly recommend that you use the requests module instead of the very bare-bones urllib2.urlopen.

Upvotes: 3

Related Questions