CODEWITHSUNDEEP
phpjsonapi
Jamie Sutton
Jamie Sutton

Reputation: 163

API call limited to 500 results - how to iterate through to get all results?

I am developing a web app which iterates through API results to insert into a MySQL database.

The API I am using limits records returned to a maximum of 500 per call.

They have a parameter where you can choose which "page" to return results from, so for 1504 (current number of records) records, I could call page 2 and return results from 500-999.

I am semi novice at best, and this is my first foray into working with APIs...I am struggling to get my head around how I could gather all 1504 results.

The API documentation for this is: https://developer.trademe.co.nz/api-reference/search-methods/rental-search/

I can use the "page" parameter to choose which page of results I want.

$url = "https://api.tmsandbox.co.nz/v1/Search/Property/Rental.json";

$headr = array();
$headr[] = 'Content-length: 0';
$headr[] = 'Authorization: OAuth oauth_consumer_key="xxx", oauth_signature_method="PLAINTEXT", oauth_signature="xxx"';

$curl = curl_init();
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_TIMEOUT, 120);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 120);
curl_setopt($curl, CURLOPT_HEADER, 0);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_HTTPHEADER,$headr);
$pageSize = 10;
$data = array(
    "rows" => $pageSize,
    "sort_order" => "ExpiryDesc",
    "page" => 1
);
$totalPages = PHP_INT_MAX;
$result = [];
do {
    curl_setopt($curl, CURLOPT_URL, $url."?".http_build_query ($data));
    $resulta = curl_exec($curl);

    $array = json_decode($resulta, true);
    array_merge($result, $array);
    $data['page']++;
    if ( $totalPages == PHP_INT_MAX )   {
        $totalPages = floor($array['TotalCount']/$pageSize);
    }

}
while ( $totalPages < $data['page']);

$properties = $array['List'];
$prop_count = $array['TotalCount'];
curl_close($curl);

This is the code I am using to return the 500 results. $prop_count holds the total number of rows (1504) and $pages holds the number of pages (in this case, 3.08).

Can anyone please give me a head start on the logical order to implement some kind of loop to return all results? I'm sitting here and racking my brain trying to get started.

Upvotes: 2

Views: 3342

Answers (1)

Nigel Ren
Nigel Ren

Reputation: 57121

Although not able to test this, the principle here is to keep on making requests until there are no more pages. The first time round the loop it will work out the number of pages(using TotalCount from the returned content / pageSize - using floor() so that 3.1 becomes 3). Then at the end of the loop it checks if the current page is > total pages.

$curl = curl_init();
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_TIMEOUT, 120);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 120);
curl_setopt($curl, CURLOPT_HEADER, 0);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_HTTPHEADER,$headr);
$pageSize = 500;
$data = array(
    "rows" => $pageSize,
    "sort_order" => "ExpiryDesc",
    "page" => 1
);
$totalPages = PHP_INT_MAX;
$result = [];
do {
    curl_setopt($curl, CURLOPT_URL, $url."?".http_build_query ($data));
    $resulta = curl_exec($curl);

    $array = json_decode($resulta, true);
    $result[] = $array['List'];
    $data['page']++;
    if ( $totalPages == PHP_INT_MAX )   {
        $totalPages = floor($array['TotalCount']/$pageSize);
    }

}
while ( $totalPages > $data['page']);
curl_close($curl);

// Loop through the pages of results and in each page each listing
foreach ( $result as $page )    {
    foreach ( $page as $item ){
        // Process each item
        echo $item['ListingId'].PHP_EOL;
    }
}

Upvotes: 2

Related Questions