CODEWITHSUNDEEP
iosobjective-cjsoncore-datafile-io
Jeef
Jeef

Reputation: 27285

Speed up writing a ton of JSON data to disk

I have a loop of maybe 40k records that basically calls. It seems that the writing to file is REALLY SLOW. If I Iterate through the loop its almost instant so I realize its not the core-data iteration that is slow is the file writing process. Is there a better way to stream data to a file than what I'm doing here? 

#ifndef jsonAppend
#define jsonAppend(X) [outStream write:[[X dataUsingEncoding:NSUTF8StringEncoding ] bytes] maxLength:[X lengthOfBytesUsingEncoding:NSUTF8StringEncoding]]
#endif



NSOutputStream *outStream = [NSOutputStream outputStreamToFileAtPath:tempFilePath append:NO];

    dispatch_async(backgroundQueue, ^{

// Open an output stream to write to.
[outStream open];

// Other code ... blablabla

for (AHRSMessage *msg in results)
{

    @autoreleasepool
    {

        [NSJSONSerialization writeJSONObject:msg.attributesAsDictionary toStream:outStream options:0 error:&error];
        jsonAppend(@",\n");

        i++;  // MessageCounter


        // Update UI only sometimes
        if (i % 100)
        {

            dispatch_async(dispatch_get_main_queue(), ^
            {
                @autoreleasepool {


                    float pct = i / recordCount ;
                    NSString *pctStr = [NSString stringWithFormat:@"%02.0f%%", pct * 100.0];
                    [[weakSelf percentLabel] setText:pctStr];
                    [[weakSelf parsingProgress] setProgress:pct animated:YES];

                    /* - Animate inner circle on units of 1% - */
                    int singPctMsgCount = recordCount / 100;
                    float fastParse = (i % singPctMsgCount) / (float)singPctMsgCount;
                    [[weakSelf fastParsingProgress] setProgress:fastParse animated:YES]     ;

                    [weakSelf updateUI];

                }
            });

        }

    }
} // end for loop

});

Upvotes: 0

Views: 599

Answers (1)

Abhi Beckert
Abhi Beckert

Reputation: 33369

Since it's only 18MB, just serialise it to an NSMutableData object, then write that to disk.

That should be extremely fast. NSMutableData can trivially handle amounts of data that do not even fit in flash memory (assuming you have a 64 bit processor), let alone in RAM on an iOS device.

Like this:

dispatch_async(backgroundQueue, ^{

  NSMutableData *outData = [NSMutableData data];

  // Other code ... blablabla

  for (AHRSMessage *msg in results)
  {

      @autoreleasepool
      {
          [outData appendData:[NSJSONSerialization dataWithJSONObject:msg.attributesAsDictionary options:0 error:&error];

          i++;  // MessageCounter


          // Update UI only sometimes
          if (i % 100)
          {

              dispatch_async(dispatch_get_main_queue(), ^
              {
                  @autoreleasepool {

                      ... update progress bar ...

                  }
              });

          }

      }
  } // end for loop

  [outData writeToURL:outURL atomically:YES];
});

Also, I would not use if (i % 100) to decide it's time to update the progress bar. Instead I would use:

CFTimeInterval lastProgressUpdate = CACurrentMediaTime();

for (  ... ) {
  ...

  if (CACurrentMediaTime() - lastProgressUpdate > 0.02) { // 1/60th of a second. Which is the refresh rate of most LCD screens
    ... update progress bar ....

    lastProgressUpdate = CACurrentMediaTime()
  }
}

Upvotes: 1

Related Questions