Core Data importation very slow

Question

Im developing an app which needs to import data in different tables from Json. I use a generic function for all the tables. The problem is that the importation extremely slow when the table have many records.

The parseJSON function:

-(void)parseJSON:(NSArray *)json entity:(NSString *)entityName{

[json enumerateObjectsUsingBlock:^(id obj, NSUInteger idx, BOOL *stop) {
    NSArray *primaryKeyNameJSON = [self getPrimaryKeyNameForEntityInJSON:entityName];
    NSArray *primaryKey = [obj objectsForKeys:primaryKeyNameJSON notFoundMarker:@""];
    NSArray *primaryKeyName = [self getPrimaryKeyNameForEntity:entityName];

    NSDictionary *primaryKeysAndValues = [NSDictionary dictionaryWithObjects:primaryKey forKeys:primaryKeyName];

    NSManagedObject *managedObj = [self getManagedObjectForEntity:entityName primaryKey:primaryKeysAndValues jsonEntity:entityName json:obj];

    NSDictionary *allAttributes = [[managedObj entity] attributesByName];
    NSArray *attributesKeys = [allAttributes allKeys];

    NSDictionary *allRelationships = [[managedObj entity] relationshipsByName];
    NSArray *relationshipsKeys = [allRelationships allKeys];

    NSArray *keys = [obj allKeys];
    for (NSString *jsonKey in keys) {

        NSString *dbKey = [self getDBKeyForJSONKey:jsonKey entity:entityName];
        if ([attributesKeys containsObject:dbKey]) {
            //Attribute
            NSAttributeDescription *oneAttribute = [allAttributes objectForKey:dbKey];
            if ([oneAttribute attributeType] == NSDateAttributeType){
                [managedObj setValue:[[self dateFormatterJSon] dateFromString:[obj objectForKey:jsonKey]] forKey:dbKey];
            }else if ([oneAttribute attributeType] == NSInteger32AttributeType || [oneAttribute attributeType] == NSFloatAttributeType){
                [managedObj setValue:[[self numberFormatterJSon] numberFromString:[obj objectForKey:jsonKey]] forKey:dbKey];
            }else if ([oneAttribute attributeType] == NSStringAttributeType)
                [managedObj setValue:[obj objectForKey:jsonKey] forKey:dbKey];
            else if ([oneAttribute attributeType] == NSBooleanAttributeType) {
                if ([[obj objectForKey:jsonKey]  isEqual: @"-1"] || [[obj objectForKey:jsonKey]  isEqual: @"1"]){
                    [managedObj setValue:[NSNumber numberWithBool:YES] forKey:dbKey];
                }else{
                    [managedObj setValue:[NSNumber numberWithBool:NO] forKey:dbKey];
                }
            }
        }
        else if ([relationshipsKeys containsObject:dbKey]) {
            //Relationship
            NSRelationshipDescription *oneRelationship = [allRelationships objectForKey:dbKey];
            NSString *destRelationshipEntityName = [[oneRelationship destinationEntity] name];

            NSArray *destRelationshipPrimaryKeyName = [self getPrimaryKeyNameForEntity:destRelationshipEntityName];
            NSArray *destRelationshipPrimaryKeyValues = [[NSArray alloc] init];
            destRelationshipPrimaryKeyValues = [obj objectsForKeys:[self getPrimaryKeyNameForEntity:destRelationshipEntityName inJSONEntity:entityName attrName:jsonKey] notFoundMarker:@""];
            NSDictionary *destRelPrimaryKeysAndValues = [NSDictionary dictionaryWithObjects:destRelationshipPrimaryKeyValues forKeys:destRelationshipPrimaryKeyName];

            NSManagedObject *destManagedObject = [self getManagedObjectForEntity:destRelationshipEntityName primaryKey:destRelPrimaryKeysAndValues jsonEntity:entityName json:obj];

            [managedObj setValue:destManagedObject forKey:dbKey];
    }
}];
}

The getManagedObjectForEntity function (here we look for the managed object in the database and create it if not exists):

-(NSManagedObject *)getManagedObjectForEntity:(NSString *)entityName primaryKey:(NSDictionary *)key jsonEntity:(NSString *)jsonEntity json:(NSDictionary *)json{

NSEntityDescription *entityDescription = [NSEntityDescription entityForName:entityName inManagedObjectContext:context];
NSFetchRequest *request = [[NSFetchRequest alloc] init];
[request setEntity:entityDescription];
NSArray *primaryKeyName = [key allKeys];
NSMutableArray *subpredicates = [NSMutableArray array];
NSError *error = nil;
NSArray *managedObjectArray = [[NSArray alloc] init];
for (NSString *k in primaryKeyName) {
    NSPredicate *predicate;
    NSString *value = [key objectForKey:k];
    predicate = [NSPredicate predicateWithFormat:@"(%K == %@)",k,value];

    [subpredicates addObject:predicate];
}
NSCompoundPredicate *predicate = [NSCompoundPredicate andPredicateWithSubpredicates:subpredicates];

[request setPredicate:predicate];

NSError *error = nil;
managedObjectArray = [context executeFetchRequest:request error:&error];

if (managedObjectArray == nil)
{
    // Deal with error...
}


if ([managedObjectArray count] > 0)
    return [managedObjectArray firstObject]; //Object exists

//If object is not found lets create it

NSManagedObject *managedObject = [NSEntityDescription insertNewObjectForEntityForName:entityName inManagedObjectContext:context];

NSDictionary *allAttributes = [entityDescription attributesByName];
NSDictionary *allRelationships = [entityDescription relationshipsByName];

NSArray *allAttributesName = [allAttributes allKeys];

for (NSString *k in primaryKeyName) {
    if ([allAttributesName containsObject:k]) {
        //Key is an attribute
        [managedObject setValue:[key objectForKey:k] forKey:k];
    }
    else {

        //Key is a relationship
        NSString *relationshipKey = [[k componentsSeparatedByString:@"."] firstObject];
        NSRelationshipDescription *oneRelationship = [allRelationships objectForKey:relationshipKey];

        NSString *destEntityName = [[oneRelationship destinationEntity] name];
        NSArray *destEntityPrimaryKey = [self getPrimaryKeyNameForEntity:destEntityName];

        NSArray *destEntityPrimaryKeyValues = [json objectsForKeys:[self getPrimaryKeyNameForEntity:destEntityName inJSONEntity:jsonEntity attrName:relationshipKey] notFoundMarker:@""];

        NSManagedObject *destManagedObj = [self getManagedObjectForEntity:destEntityName primaryKey:[NSDictionary dictionaryWithObjects:destEntityPrimaryKeyValues forKeys:destEntityPrimaryKey] jsonEntity:jsonEntity json:json];

        [managedObject setValue:destManagedObj forKey:relationshipKey];
    }
}

return managedObject;
}

Finally i use the [context save:&error] function when all the JSON data has been imported.

I have tried saving each table in an array and looking for the managed objects into the array, but the big number of data in each table causes memory problems. Is there a better way to import all this data faster? Thanks

Answer 1

The find and create pattern is always going to be expensive. However, your design is hitting the disk for each individual record. That is wasteful and most likely the core of your slowness.

If you are using iOS 8 or earlier then you should be using KVC to get all of the primary keys at once and fetching the existing records all at once (or at least their NSManagedObjectID) and then you can make the decision to do an insert or update.

If you are using iOS 9 or newer it is even easier as you can set your primary key as a constraint in the persistent store (assuming you are using SQLite) and then just insert all the records and let Core Data solve the insert vs. update problem.

As for memory pressure, you most likely want to break the job down and save a few times during the import. Whenever you save you can then call -reset on the context to ask it to drop its objects from memory. That will help you keep the pressure down.

Lastly, use Instruments. While the general advice I gave you above will help, you can fine tune the import by using the time profiler and looking for hot spots in the code yourself.