CODEWITHSUNDEEP
iosobjective-csqlitecore-data
stuckj
stuckj

Reputation: 996

Core data migration without reading all of the data into memory?

We have a core data DB (sqlite store) which, for some users, is about 100-150 MB. I wouldn't think that would be too big for a storage system to deal with (even on a mobile device), but we've found that with that size core data DB, ANY lightweight migration takes ~10+ seconds to complete. Even something as simple as adding a completely new independent entity (not related to any other entity). With raw sqlite this would be a create table statement. So, my question is whether anyone else has seen this and, if so, have they found a workaround to make such simple migrations faster? Specifically, I'm looking for a way to handle adding a new independent entity to an existing core data DB that's ~100-150 MB and have it be quick (i.e., under 5 seconds).

I believe that core data migrations ALWAYS have to read all of the data from the source and write it all to a destination for a migration (which is terrible BTW), but I'm hoping someone can prove me wrong. :) I couldn't find any way to do it with a custom migration either.

I've considered munging the DB with sql directly to basically make the model look like what CoreData would expect (I've done stuff like this to manually "downgrade" a core data DB for debugging purposes), but we really want to avoid doing something like that in production.

UPDATE:

For reference, this is the current approach we are taking. This is not a generic solution, but will work for our use case. Unless I get a better answer I'll add this as an answer at some point in the future and accept it.

We're going to deal with this by essentially making the DB smaller. There are 2 out of 15 entities that take up the bulk of the space in the DB (~95%). We're going to create completely separate data models each with one of those entities. This is done without changing the main model at all (hence, no core data migration). We'll then make a task that runs with background priority in GCD that, if any of those entities are found in the main DB, they're moved to the appropriate separate DB and removed from the main DB. This is done in batches with some sleeps between batches so it's less resource intensive and doesn't affect normal app operation. We'll modify the code that accesses those entities to try to get them from the new DB and fall back to the main DB if they're not in there.

In a future update after we find that all, or at least most, of our users have updated their data in the new DBs we'll drop those entities from the main DB.

This leaves us with a small main DB that can have migrations applied quickly and two large DBs that have migrations done more slowly. Those large DBs, in our case, should change less often (maybe never?) and even if they do change there are limited places in the app that require them so we can work around it in the UI (e.g., report some feature as unavailable until we move data).

Upvotes: 1

Views: 415

Answers (1)

Hal Mueller
Hal Mueller

Reputation: 7655

A 10-20 second delay for an update to a huge dataset seems perfectly reasonable to me. Just don't do it on the main thread.

This means you'll have to modify the boilerplate Core Data stack setup that you get in the usual Xcode templates. Instead of always setting up the stack on the main thread at launch time, check to see if migration is needed. If so, put up appropriate UI, do the migration in a background thread, and be ready to invoke beginBackgroundTaskWithExpirationHandler: if needed.

Upvotes: 2

Related Questions