CODEWITHSUNDEEP
ioscore-dataobjective-c-blocks
Sarabjit Singh
Sarabjit Singh

Reputation: 1824

Saving 1 million records to core data taking so much time

I am saving 1 million records from CSV to the core database. It takes near about 35min to save all the records.Is there anyway to reduce the insertion time. Here is the my code which i used to insert the records :

 NSString *resourceFileName = @"npidata_20140209_reduced2";
NSString *pathToFile =[[NSBundle mainBundle] pathForResource: resourceFileName ofType: @"csv"];

NSString* content = [NSString stringWithContentsOfFile:pathToFile
                                              encoding:NSUTF8StringEncoding
                                                 error:NULL];
__block int count = 0;

[content enumerateLinesUsingBlock:^(NSString *line, BOOL *stop) { NSLog(@"LINE==%@",line);

    NSArray *lineComponents=[line componentsSeparatedByString:@","];
    if(lineComponents){       
            float f=[[lineComponents objectAtIndex:0] floatValue];
            NSString *strNPI=[lineComponents objectAtIndex:0];
            NSString *strProviderLastName=[lineComponents objectAtIndex:1];
            NSString *strProviderFirstName=[lineComponents objectAtIndex:2];
            NSString *strLicenseNumber=[lineComponents objectAtIndex:6];
            NSString *strLicenseState=[lineComponents objectAtIndex:7];         
            NSManagedObject *object=[NSEntityDescription insertNewObjectForEntityForName:@"AttendeeInformation" inManagedObjectContext:[DataBaseManager sharedInstance].managedObjectContext];

            NSString *strFullName = [NSString stringWithFormat:@"%@ %@",strProviderFirstName,strProviderLastName];

            [object setValue:strNPI forKey:@"npiNumber"];
            [object setValue:strProviderFirstName forKey:@"attendeeFirstName"];
            [object setValue:strProviderLastName forKey:@"attendeeLastName"];
            [object setValue:strFullName forKey:@"attendeeName"];
            [object setValue:strLicenseNumber forKey:@"licnesedNumber"];
            [object setValue:strLicenseState forKey:@"stateLicense"];

            NSError *error;
            count++;
            if(count%100==0)
            {
                if (![[DataBaseManager sharedInstance].managedObjectContext save:&error])
                {
                    NSLog(@"Whoops, couldn't save: %@", [error localizedDescription]);
                }
                [[DataBaseManager sharedInstance].managedObjectContext refreshObject:object mergeChanges:YES];
            }
        if (count==lineComponents.count)
        {
            completionHandler();
        }
    }
}];

Upvotes: 3

Views: 1495

Answers (2)

RobP
RobP

Reputation: 9522

@Desdenova had it right. You need to get all the records in your context and save the context once at the end, or at least in very large batches. Under the API, sqlite is doing a separate transaction for each context save which is very expensive. If you do all the operations as a single transaction it's orders of magnitude faster. Do a test run where you comment out:

            if (![[DataBaseManager sharedInstance].managedObjectContext save:&error])
            {
                NSLog(@"Whoops, couldn't save: %@", [error localizedDescription]);
            }

and add these same 3 lines after the loop.

Upvotes: 2

Z S
Z S

Reputation: 7479

You should run Instruments with Time Profile and Core Data to see what your bottlenecks are. It's the best way to validate your performance issues. Core Data might or might not be your problem.

Upvotes: 3

Related Questions