CODEWITHSUNDEEP
apirestparse-platform
James
James

Reputation: 215

Import 140k objects to Parse.com

I am trying to populate a Data Class in Parse with 140k objects. Because I have GeoPoints, I can not import via the web interface. The REST API they suggest does not allow you to create multiple objects with one API call. So, it looks like I will be making 140k API calls.

Right now, I have a file with 140k rows of properly formed JSON, waiting to be imported into Parse.

I have tried doing this with curl in PHP on my webserver. I got 5-10k objects imported before the script stops running. I tried adding this to the PHP file-

ignore_user_abort(true);
set_time_limit(0);

But this didn't work. So I thought maybe I'd have more control in Javascript, again 5-10k objects imported before crashing. I added a timeout pause every 50 objects, and it still crashed. I've tried waiting for a success response before making the next API call, or running them all concurrently, and it doesn't make a difference.

Anyways, my question is how to most efficiently make 140k REST API calls, without crashing.

Upvotes: 2

Views: 1960

Answers (2)

hank
hank

Reputation: 3768

Even if you can import complex layouts using the JSON importer, that importer have som issues with large files

You can create multiple objects in one call. Use /1/batch, See example usage below;

{
    "requests": [{
        "method": "POST",
        "path": "/1/classes/YOURCLASS",
        "body": {
            "your": "parameters",
            "live": "here"
        }
    }, {
        "method": "POST",
        "path": "/1/classes/YOURCLASS",
        "body": {
            "second": "request",
            "parameters": "here"
        }
    }]
}

I've successfully imported around 10k rows like this without any problem, I did run it through the terminal however using PHP and cURL, here's the cURL call for reference;

$headers = array(
    'X-Parse-Application-Id: APPID',
    'X-Parse-REST-API-Key: RESTKEY',
    'Content-type: application/json'
);

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.parse.com/1/batch');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_POSTFIELDS, $json);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$ret = curl_exec($ch);

CURLOPT_RETURNTRANSFER is not necessary if you are not interested in if all went well or not, aka "fire and forget".

Good luck!

Upvotes: 1

Héctor Ramos
Héctor Ramos

Reputation: 9258

You can import geopoints using Parse's JSON importer (instead of CSV). See Importing Data for details.

Upvotes: 2

Related Questions