CODEWITHSUNDEEP
javagoogle-app-engine
Arun
Arun

Reputation: 43

How to upload bulk data to Google Servers using Google App Engine running on Java?

I am not able to figure out how to upload bulk data to the Google's servers bypassing the 10mb upload limit and 30 sec session timeout. I want to design an application that takes my standard SQL data and pushes it to the Google's servers.

I might sound naive but your help is most valuable for my project.

Upvotes: 3

Views: 2128

Answers (2)

Abel Callejo
Abel Callejo

Reputation: 14919

I wanna do the same thing also. So, here's my naivest concept to achieve the goal.

Web Server Preparation

  1. Create a servlet that will receive the uploaded data (e.g. for data type XML, JSON)

    (optional) store it as Blobstore

  2. Parse the data using JAXB/JSoup and/or GSON
  3. Dynamically interpret the data structure
  4. Store it using Datastore/

Client Uploader Preparation

  1. Using a local computer, create a Java/C++/PHP script that generates XML/JSON files and store it locally
  2. Create a shell script (linux) or batch file (windows) to programatically upload the files using cURL.

Please drop a comment to this one if you have better idea guys.

Upvotes: 0

Nick Johnson
Nick Johnson

Reputation: 101139

There's not currently a native Java bulkloader, so what you need to do is use the Python one. The process goes like this:

First, you'll need to download the Python SDK and extract it. Then, create an empty directory, and in it create a file called app.yaml, containing the following:

application: yourappid
version: bulkload
runtime: python
api_version: 1

handlers:
- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

Now, run "appcfg.py update yourdir" from the Python SDK, and enter your credentials when prompted. appcfg will upload a new version of your app, which will run side-by-side with your main version, and allow you to bulkload.

Now, to do the actual bulkloading, you need to use the Python Bulkloader. Follow the instructions here. You'll need to know a (very) little bit of Python, but it's mostly copy-and-paste. When you're done, you can run the bulkloader as described in the article, but add the "-s bulkload.latest.yourapp.appspot.com" argument to the command line, like this:

appcfg.py upload_data --config_file=album_loader.py --filename=album_data.csv --kind=Album -s bulkload.latest.yourapp.appspot.com <app-directory>

Finally, to load data directly from an SQL database instead of from a CSV file, follow the instructions in my blog post here.

Upvotes: 8

Related Questions