CODEWITHSUNDEEP
pythonparameterspython-requests
Provisional.Modulation
Provisional.Modulation

Reputation: 724

Specify page_count parameter from API request

I'm downloading json data from an API, but I am having trouble setting a parameter. Specifically, I'd like create the option of setting how many number of pages I'd like to download from the API. My code looks something like this:

# Libraries
import json, requests

# Define API, endpoint, and page number
url = 'https://api-v2.themuse.com/jobs?page=1'

# Set page count parameter
params = {'page_count': 10}

# Request data from API and store into object
resp = requests.get(url=url, params=params)

# Save data as JSON
data = json.loads(resp.text)

However, the 'params' function doesn't set the number of pages I want to download: Instead of 10 pages, I download 100 pages. Any thoughts? If it helps, the JSON structure looks something like this:

{
  "page": 1,
  "page_count": 100,
  "took": 10,
  "timed_out": false,
  "total": 35210
}

Upvotes: 0

Views: 1069

Answers (1)

Harald Nordgren
Harald Nordgren

Reputation: 12399

Looking at the API specification, page_count is an output parameter and not something you specify. Response data is paginated with 20 results per page.

You are requesting page=1 (the index starts from page 0) and getting back 20 results. To get all the data for the first 10 pages, do something like this:

jobs_url = 'https://api-v2.themuse.com/jobs'

for i in range(11):
    results_data = json.loads(requests.get(
        url=jobs_url,
        params={'page': i}
    ).text)['results']

    #do something with the data

Upvotes: 1

Related Questions