CODEWITHSUNDEEP
phpcurlsalesforce
user1274820
user1274820

Reputation: 8144

Combining Large JSON Sent as Multiple Chunks

I'm working with cURL and PHP to access a Salesforce API and I've got the majority of it working.

When you request a large number of records, Salesforce will break the data into chunks of 2000.

Basically, I have three different, large JSON chunks with arrays of records and I'm wondering what the best way to combine them into one large object is.

The objects look like this when there is more data.

It gives you a nextRecordsUrl which you hit to get the next chunk:

Array
(
    [totalSize] => 5000
    [done] => 
    [nextRecordsUrl] => /services/data/v20.0/query/somelongurlstring
    [records] => Array
        (
            [0] => Array
                (
                    [attributes] => Array
                        (
                            [type] => Custom_Type__c
                            [url] => sobjects/Custom_Type__c/thetypeid
                        )

                    [Requested_Prop__c] => someid
                )

            [1] => Array
                ( [...]

My curl code seems to be working - it's basically this:

curl_setopt($ch2, CURLOPT_URL, "https://my.instance.salesforce.com/services/data/v20.0/query/?q=" . urlencode($_GET["q"]));
curl_setopt($ch2, CURLOPT_HTTPHEADER, array("Content-Type: application/json", "Authorization: Bearer $token"));
curl_setopt($ch2, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch2, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, 1);
$isDone = false;
while(!$isDone) {
    $qresponse = curl_exec($ch2);
    echo "<pre>";
    $theJson = json_decode($qresponse, true);
    echo print_r($theJson);
    echo "</pre>";
    $isDone = $theJson["done"] == true;
    if(!$isDone)
        curl_setopt($ch2, CURLOPT_URL, "https://my.instance.salesforce.com" . $theJson["nextRecordsUrl"]);
}
curl_close($ch2);

The problem is that the output is actually multiple json objects.

For example, the second fetch request starts like this:

Array
(
    [totalSize] => 5000
    [done] => 
    [nextRecordsUrl] => /services/data/v20.0/query/somelongurlstring
    [records] => Array
        (
            [0] => Array
                (
                    [attributes] => Array
                        (
                            [type] => Custom_Type__c
                            [url] => sobjects/Custom_Type__c/thetypeid
                        )

                    [Requested_Prop__c] => someid
                )

I thought the easiest way to do this is probably to simply loop through the records and append them to some json object I create in PHP, but my concern was just the amount of memory that might take if these fetch requests get very large.

Is there a "right way" to do this?

Thanks for your help

Upvotes: 0

Views: 369

Answers (1)

user1274820
user1274820

Reputation: 8144

I ended up just appending the data myself:

$isDone = false;
$finalJson = array();
while(!$isDone) {
    $qresponse = curl_exec($ch2);
    $theJson = json_decode($qresponse, true);
    $isDone = $theJson["done"] == true;
    foreach($theJson["records"] as $record) {  //Only return records
        $finalJson[] = $record;
    }
    if(!$isDone)
        curl_setopt($ch2, CURLOPT_URL, "https://my.instance.salesforce.com" . $theJson["nextRecordsUrl"]);
}
echo "<pre>" . json_encode($finalJson, JSON_PRETTY_PRINT) . "</pre>";

Upvotes: 1

Related Questions