Using a file (CSV) instead of using CoreData

Question

An app I'm building contains a catalogue of thousands of items, which need to be stored on the phone. Currently I am achieving this through CoreData, as logically it seemed like the best place to put it. I'm using GCD to run the CoreData insertion processes in the background and showing a progress bar / current percentage complete. This works as expected, however for only 5000 items, it's taking 8 minutes to complete on an iPhone 4. This application will be used on the 3GS and up, and will more likely contain 30/40 thousand items once it launches. Therefore this processing time is going to be horrifically long.

Is there any way I can use a CSV file or something to search through instead of storing each item in CoreData? I'm assuming there are some efficiency downfalls with an approach like this, but it would alleviate the excessive wait times. Unless there is another solution that would help with this problem.

Thanks.

EDIT: I'm not sure how I'd go about saving the context at the end of the entire operation, as it uses a separate context within the loop. Any suggestions for this would be very much appreciated. I've got no idea how to progress with this.

Insertion Code Being Used

- (void) processUpdatesGCD {
    NSArray *jsonArray=[NSJSONSerialization JSONObjectWithData:_responseData options:0 error:nil];
    NSArray *products = [jsonArray valueForKey:@"products"];
    NSArray *deletions;
    if ([jsonArray valueForKey:@"deletions"] == (id)[NSNull null]){
        self.totalCount = [products count];
    } else {
        deletions = [jsonArray valueForKey:@"deletions"];
        self.totalCount = [products count] + [deletions count];
    }

    self.productDBCount = 0;

    _delegate = [[UIApplication sharedApplication] delegate];
    NSManagedObjectContext *managedObjectContext = _delegate.managedObjectContext;
    self.persistentStoreCoordinator = [managedObjectContext persistentStoreCoordinator];
    _managedObjectContext = managedObjectContext;



    // Create a new background queue for GCD
    dispatch_queue_t backgroundDispatchQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0);

    for (id p in products) {

//        id product = p;

        // Dispatch the following code on our background queue
        dispatch_async(backgroundDispatchQueue,
                       ^{
                           id product = p;
                           // Because at this point we are running in another thread we need to create a
                           // new NSManagedContext using the app's persistance store coordinator

                           NSManagedObjectContext *backgroundThreadContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSConfinementConcurrencyType];
                           [backgroundThreadContext setPersistentStoreCoordinator:self.persistentStoreCoordinator];

                           NSFetchRequest *BGRequest = [[NSFetchRequest alloc] init];
                           NSLog(@"Running.. (%@)", product);
                           [BGRequest setEntity:[NSEntityDescription entityForName:@"Products" inManagedObjectContext:backgroundThreadContext]];
                           [BGRequest setIncludesSubentities:NO];

                           NSPredicate *predicate = [NSPredicate predicateWithFormat:@"codes == %@", [product valueForKey:@"product_codes"]];
                           [BGRequest setPredicate:predicate];

                           NSError *err;
                           NSArray *results = [backgroundThreadContext executeFetchRequest:BGRequest error:&err];

                           if (results.count == 0){
                               // Product doesn't exist with code, make a new product

                               NSLog(@"Product not found for add/update (%@)", [product valueForKey:@"product_name"]);

                               NSManagedObject* newProduct;
                               newProduct = [NSEntityDescription insertNewObjectForEntityForName:@"Products" inManagedObjectContext:backgroundThreadContext];

                               [newProduct setValue:[product valueForKey:@"product_name"] forKey:@"name"];
                               [newProduct setValue:[product valueForKey:@"product_codes"] forKey:@"codes"];

                               if ([product valueForKey:@"information"] == (id)[NSNull null]){
                                   // No information, NULL
                                   [newProduct setValue:@"" forKey:@"information"];
                               } else {
                                   NSString *information = [product valueForKey:@"information"];
                                   [newProduct setValue:information forKey:@"information"];

                               }

                           } else {
                               NSLog(@"Product found for add/update (%@)", [product valueForKey:@"product_name"]);
                               // Product exists, update existing product
                               for (NSManagedObject *r in results) {
                                   [r setValue:[product valueForKey:@"product_name"] forKey:@"name"];

                                   if ([product valueForKey:@"information"] == (id)[NSNull null]){
                                       // No information, NULL
                                       [r setValue:@"" forKey:@"information"];
                                   } else {
                                       NSString *information = [product valueForKey:@"information"];
                                       [r setValue:information forKey:@"information"];
                                   }

                               }

                           }



                           // Is very important that you save the context before moving to the Main Thread,
                           // because we need that the new object is writted on the database before continuing
                           NSError *error;

                           if(![backgroundThreadContext save:&error])
                           {
                               NSLog(@"There was a problem saving the context (add/update). With error: %@, and user info: %@",
                                     [error localizedDescription],
                                     [error userInfo]);
                           }


                           // Now let's move to the main thread
                           dispatch_async(dispatch_get_main_queue(), ^
                                          {
                                              // If you have a main thread context you can use it, this time i will create a
                                              // new one
//                                              NSManagedObjectContext *mainThreadContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSConfinementConcurrencyType];
//                                              [mainThreadContext setPersistentStoreCoordinator:self.persistentStoreCoordinator];

                                              self.productDBCount = self.productDBCount + 1;                                              
                                              float progress = ((float)self.productDBCount / (float)self.totalCount);
                                              int percent = progress * 100.0f;
//                                              NSNumber *progress = [NSNumber numberWithFloat:((float)self.productDBCount / (float)self.totalCount)];
                                              self.downloadUpdateProgress.progress = progress;
                                              self.percentageComplete.text = [NSString stringWithFormat:@"%i", percent];
                                              NSLog(@"Added / updated product %f // ProductDBCount: %i // Percentage progress: %i // Total Count: %i", progress, self.productDBCount, percent, self.totalCount);
                                              if (self.productDBCount == self.totalCount){
                                                  [self updatesCompleted:[jsonArray valueForKey:@"last_updated"]];
                                              }

                                          });
                       });

    }


    if ([deletions count] > 0){
        for (id d in deletions){
            dispatch_async(backgroundDispatchQueue,
                           ^{
                               id deleted = d;
                               // Because at this point we are running in another thread we need to create a
                               // new NSManagedContext using the app's persistance store coordinator

                               NSManagedObjectContext *backgroundThreadContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSConfinementConcurrencyType];
                               [backgroundThreadContext setPersistentStoreCoordinator:self.persistentStoreCoordinator];

                               NSFetchRequest *BGRequest = [[NSFetchRequest alloc] init];
//                               NSLog(@"Running.. (%@)", deleted);
                               [BGRequest setEntity:[NSEntityDescription entityForName:@"Products" inManagedObjectContext:backgroundThreadContext]];
                               [BGRequest setIncludesSubentities:NO];

                               NSPredicate *predicate = [NSPredicate predicateWithFormat:@"codes == %@", [deleted valueForKey:@"product_codes"]];
                               [BGRequest setPredicate:predicate];

                               NSError *err;
                               NSArray *results = [backgroundThreadContext executeFetchRequest:BGRequest error:&err];

                               if (results.count == 0){
                                   // Product doesn't exist with code, make a new product

                                   NSLog(@"Product not found, can't delete.. %@", [deleted valueForKey:@"product_name"]);

                               } else {
                                   NSLog(@"Product found, deleting: %@", [deleted valueForKey:@"product_name"]);
                                   // Product exists, update existing product
                                   for (NSManagedObject *r in results) {
                                       [backgroundThreadContext deleteObject:r];
                                   }

                               }



                               // Is very important that you save the context before moving to the Main Thread,
                               // because we need that the new object is writted on the database before continuing
                               NSError *error;

                               if(![backgroundThreadContext save:&error])
                               {
                                   NSLog(@"There was a problem saving the context (delete). With error: %@, and user info: %@",
                                         [error localizedDescription],
                                         [error userInfo]);
                               }


                               // Now let's move to the main thread
                               dispatch_async(dispatch_get_main_queue(), ^
                                              {

                                                  self.productDBCount = self.productDBCount + 1;
                                                  float progress = ((float)self.productDBCount / (float)self.totalCount);
                                                  int percent = progress * 100.0f;
                                                  //                                              NSNumber *progress = [NSNumber numberWithFloat:((float)self.productDBCount / (float)self.totalCount)];
                                                  self.downloadUpdateProgress.progress = progress;
                                                  self.percentageComplete.text = [NSString stringWithFormat:@"%i", percent];
                                                  NSLog(@"Deleted product %f // ProductDBCount: %i // Percentage progress: %i // Total Count: %i", progress, self.productDBCount, percent, self.totalCount);
                                                  if (self.productDBCount == self.totalCount){
                                                      [self updatesCompleted:[jsonArray valueForKey:@"last_updated"]];
                                                  }

                                                  /*
                                                   *
                                                   * Change the completion changes to a method. Check to see if the total number of products == total count. If it does, run the completion method. 
                                                   *
                                                   */

                                              });
                           });
        }
    }


}

Put the IF inside the dispatch, run one save at the end

    // Create a new background queue for GCD
dispatch_queue_t backgroundDispatchQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0);
    //        id product = p;

    // Dispatch the following code on our background queue
  dispatch_async(backgroundDispatchQueue,
    ^{

      NSManagedObjectContext *backgroundThreadContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSConfinementConcurrencyType];
      [backgroundThreadContext setPersistentStoreCoordinator:self.persistentStoreCoordinator];

      for (id p in products) {
        id product = p;
        // Because at this point we are running in another thread we need to create a
        // new NSManagedContext using the app's persistance store coordinator



        NSFetchRequest *BGRequest = [[NSFetchRequest alloc] init];
        NSLog(@"Running.. (%@)", product);
        [BGRequest setEntity:[NSEntityDescription entityForName:@"Products" inManagedObjectContext:backgroundThreadContext]];
        [BGRequest setIncludesSubentities:NO];

        NSPredicate *predicate = [NSPredicate predicateWithFormat:@"codes == %@", [product valueForKey:@"product_codes"]];
        [BGRequest setPredicate:predicate];

        NSError *err;
        NSArray *results = [backgroundThreadContext executeFetchRequest:BGRequest error:&err];

        if (results.count == 0){
        // Product doesn't exist with code, make a new product

          NSLog(@"Product not found for add/update (%@)", [product valueForKey:@"product_name"]);

          NSManagedObject* newProduct;
          newProduct = [NSEntityDescription insertNewObjectForEntityForName:@"Products" inManagedObjectContext:backgroundThreadContext];

          [newProduct setValue:[product valueForKey:@"product_name"] forKey:@"name"];
          [newProduct setValue:[product valueForKey:@"product_codes"] forKey:@"codes"];

          if ([product valueForKey:@"information"] == (id)[NSNull null]){
          // No information, NULL
            [newProduct setValue:@"" forKey:@"information"];
          } else {
            NSString *information = [product valueForKey:@"information"];
            [newProduct setValue:information forKey:@"information"];

          }

        } else {
          NSLog(@"Product found for add/update (%@)", [product valueForKey:@"product_name"]);
          // Product exists, update existing product
          for (NSManagedObject *r in results) {
            [r setValue:[product valueForKey:@"product_name"] forKey:@"name"];

            if ([product valueForKey:@"information"] == (id)[NSNull null]){
      // No information, NULL
              [r setValue:@"" forKey:@"information"];
            } else {
              NSString *information = [product valueForKey:@"information"];
              [r setValue:information forKey:@"information"];
            }

          }

        }



      // Is very important that you save the context before moving to the Main Thread,
      // because we need that the new object is writted on the database before continuing


      // Now let's move to the main thread
        dispatch_async(dispatch_get_main_queue(), ^
        {
      // If you have a main thread context you can use it, this time i will create a
      // new one
      //                                              NSManagedObjectContext *mainThreadContext = [[NSManagedObjectContext alloc] initWithConcurrencyType:NSConfinementConcurrencyType];
      //                                              [mainThreadContext setPersistentStoreCoordinator:self.persistentStoreCoordinator];


          self.productDBCount = self.productDBCount + 1;
          float progress = ((float)self.productDBCount / (float)self.totalCount);
          int percent = progress * 100.0f;
      //                                              NSNumber *progress = [NSNumber numberWithFloat:((float)self.productDBCount / (float)self.totalCount)];
          self.downloadUpdateProgress.progress = progress;
          self.percentageComplete.text = [NSString stringWithFormat:@"%i", percent];
          NSLog(@"Added / updated product %f // ProductDBCount: %i // Percentage progress: %i // Total Count: %i", progress, self.productDBCount, percent, self.totalCount);

          NSDate *currentProcessedDate = [NSDate date];
          NSTimeInterval timeSinceStarted = [currentProcessedDate timeIntervalSinceDate:self.startProcessing];
          NSInteger remainingProcesses = self.totalCount - self.productDBCount;
          float timePerProcess = timeSinceStarted / (float)self.productDBCount;
          float remainingTime = timePerProcess * (float)remainingProcesses;
          self.timeRemaining.text = [NSString stringWithFormat:@"ETA: %0.0f minutes", fmodf(remainingTime, 60.0f)];

          if (self.productDBCount == self.totalCount){
            [self updatesCompleted:[jsonArray valueForKey:@"last_updated"]];
          }

      /*
      *
      * Change the completion changes to a method. Check to see if the total number of products == total count. If it does, run the completion method. 
      *
      */
    });

  }

    NSError *error;

        if(![backgroundThreadContext save:&error])
        {
          NSLog(@"There was a problem saving the context (add/update). With error: %@, and user info: %@",
            [error localizedDescription],
            [error userInfo]);
        }


    });

Answer 1

Ok, here's your problem.

Every time you insert a record, you do a save operation to the context. Now, don't do it, that's what takes alot of time.

Do the save operation once, in the end of the loop, not every time you insert a record.

Answer 2

In your case I would check what is really time consuming?

Is it downloading data, is it importing data to CoreData?

Where do you get data from? Do you download it or you have it in Application Bundle?

CoreData is faster then CSV file. So it wont make your app faster.

Some tricks:

• While importing data just save context at the end of the process. Do not save context in a loop.

• If you do not need to download data and can put in the bundle, you can create coredata file in the simulator, put in the bundle and copy the file on first launch. It is really much more faster then importing data.