CODEWITHSUNDEEP
javajsonhttpcsv
learner
learner

Reputation: 905

Doing HTTP post of JSON object every second

I have to do a HTTP post in java every second after building a json object. The json object is built from reading a CSV file which is huge (200Mbs+), so my problem is how do I read x number of lines build x objects and post that every second(as it is not possible to parse the whole 200mb file in less than a second) and continue reading the next x lines.

Please let me know your thoughts..

Can I use Java timer class, and keep reading the CSV file and at the same time post the json object to the server every second with the formed json?

Upvotes: 2

Views: 541

Answers (2)

atzu
atzu

Reputation: 424

I would make it depending on the file size and not depending on time. 

BufferedReader fin = null; //create it
    Gson gson=new Gson();  //Google code open source library for JSON in Java  
    ArrayList<JSONObject> jsonList=new ArrayList<JSONObject>();    
    while (((line = fin.readLine()) != null)) {
                            if ( line.length()==0 ){
                                //"Blank line;
                            }else{
                                        currJSON=loadJSON(line);//You have to load it in a Java Object

                               if ( jsonList.size()<MAX_JSON){
                               jsonList.add(currJSON);
                            }

        if (JsonList.size()==MAX_JSON){ //Define the maximum size of the list you want to post
        gson.toJson(jsonList); //Convert to JSON
        //You should post your Json with some Http Connection to your server
    jsonList.clear();

Upvotes: 0

Sean Patrick Floyd
Sean Patrick Floyd

Reputation: 298908

It is hardly possible to read, parse, convert and send a 200 MB file once per second.

So you need to change your design:

My suggestion would be to only send changed lines, something like this:

{
"1" : {"field1":"value1","field2":"value2"},
"17" : {"field1":"value1","field2":"value2"}
}

Which of course gives you new problems:

The client needs to figure out which lines have changed, and the server needs to integrate the changed lines with the existing data.

Upvotes: 1

Related Questions