Saving and fetching consumes high Memory 800MB and 100% CPU in Core Data / How to do batch insert and fetching in core data in swift4

Question

I've used core data to store 10000-20000 records. if i try to save and fetch 10000 records memory and cpu consumption was huge due to that app is getting crash in iphone 6 plus and earlier devices.

Here is the saving methods:----

   //InboxCoredata Saving Method i..(calling saving method ii)
func InboxSaveInCoreDataWith(array: [[String: AnyObject]])
{
    _ = array.map{self.InboxCreateCollectionEntityFrom(dictionary: $0)}
    do
    {
        try CoreDataStack.sharedInstance.persistentContainer.viewContext.save()
        print("Inbox Data saved Sucessfully in Coredata ")
    }
    catch let error
    {
        print(error)
    }
}

// Inbox Coredata saving method ii
func InboxCreateCollectionEntityFrom(dictionary: [String: AnyObject]) -> NSManagedObject?
{
    let context = CoreDataStack.sharedInstance.persistentContainer.viewContext
    if let inboxEntity = NSEntityDescription.insertNewObject(forEntityName: "InboxData", into: context) as? InboxData {
        inboxEntity.fileType = dictionary["FileType"] as? String
        inboxEntity.sender = dictionary["Sender"] as? String
        inboxEntity.mailPath = dictionary["MailPath"] as? String
        inboxEntity.fullMail = dictionary["FullMail"] as? NSObject
        inboxEntity.attachmentName = dictionary["AttachmentName"] as? String
        inboxEntity.size = dictionary["Size"]  as! Int32
        inboxEntity.date = dictionary["Date"] as? NSDate
        inboxEntity.dateForSearch = dictionary["DateForSearch"] as? String
        inboxEntity.inboxMail = dictionary["InboxMail"] as? String
        return inboxEntity
    }
    return nil
}

And Here is the method for fetching:----

 strongSelf.inboxDataFromCoreData(fetchLimit: 0) // Methods calling in viewdidload

//MARK: - Fetching inboxData from Coredata
func inboxDataFromCoreData(fetchLimit :Int)
{
    var inboxCoredataFetch = [[String : AnyObject]]()
    let context = CoreDataStack.sharedInstance.persistentContainer.viewContext
    let fetchRequest = NSFetchRequest<NSFetchRequestResult>(entityName:  String(describing: InboxData.self))

    do {
        fetchRequest.sortDescriptors = [NSSortDescriptor(key: "date", ascending: false)]
        fetchRequest.fetchLimit = fetchLimit
        let results = try context.fetch(fetchRequest) as! [InboxData]
        print("inbox_coredata:\(results .count)")
        for data in results
        {
            let sender = data.sender
            let mailPath = data.mailPath
            let fileType = data.fileType
            let fullMail = data.fullMail
            let attachmentName = data.attachmentName
            let size = data.size
            let date = data.date
            let dateForsearch = data.dateForSearch
            let inboxMail = data.inboxMail

            inboxCoredataFetch.append(["Sender" : sender as AnyObject,"MailPath": mailPath! as AnyObject, "FileType":fileType as AnyObject, "FullMail":fullMail as AnyObject, "AttachmentName": attachmentName as AnyObject, "Size":size as AnyObject,"Date": date as AnyObject,"DateForSearch" :dateForsearch as AnyObject,"InboxMail":inboxMail as AnyObject])
        }



    }
    catch let err as NSError {
        print(err.debugDescription)
    }

    var sortingdata = inboxCoredataFetch as Array
    let mailBoxSortDescriptor = NSSortDescriptor(key: "Date", ascending:false, selector: nil)
    let dateDescriptor = NSSortDescriptor(key: "AttachmentName", ascending: true, selector: #selector(NSString.caseInsensitiveCompare))
    sortingdata = ((sortingdata as NSArray).sortedArray(using: [ mailBoxSortDescriptor,dateDescriptor]) )  as! [[String : AnyObject]]
    inboxTotalMailData = sortingdata

    if appDataLoadingFirst == true
    {
    appDataLoadingFirst = false
    displayTotalData = sortingdata
    DispatchQueue.main.async {
        self.hideActivityIndicator()
        self.collectionView.reloadData()
    }
    }

}

Core data structure is like this:::---

i've too much confusion questions on these.

1. Core data is not good for storing 20000 records??
2. Do i need to refreshObject(object,mergeChnage:bool) for huge data everytime i've to refresh object as like that.
3. Instruments indicates memory leak in fetching results at let results = try context.fetch(fetchrequest.....line why??
4. Do i need to save and fetch data in batches will that increases app performence and memory reduce??
5. Why CPU indicates 100% sometimes ??
6. if i display 10000 records in collection view(data loading from arrray) causes any issue?? if yes what kind of issue??
7. Need your valuable suggesstion and help to make me perfect????

Answer 1

You're using a lot of memory and CPU time because:

• When you're creating new InboxData entries, you go create one for every single entry in the array before you save changes. Your code means that all of those must be in memory at the same time. If you have thousands of entries, that's a lot of memory.
• When you fetch entries, you fetch every single entry every time. Again, your code means that you must have all of them in memory at the same time. And again, lots of entries mean lots of memory. One of the main reasons to use Core Data is that it's almost always possible to load only a subset of your data into memory, instead of everything. You're loading everything.
• You copy a bunch of attributes from managed objects into dictionaries-- so now you have two copies of the data in memory instead of one.
• Your code that fetches data sorts the data in memory to create sortingdata. You already have a ton of objects loaded into memory; now you're doing a ton of work to sort them all. This will be at least part of the reason you peg the CPU at 100%.
• After sorting the data you assign the result to inboxTotalMailData and to displayTotalData, which are defined somewhere outside this function. This means that all of the data in sortingdata remains in memory even after this function finishes.

Some things that would help:

• When saving data, save at regular intervals-- every 50 entries, or every 100, or whatever number gives good results. Do not attempt to create many thousands of objects and keep them all in memory.
• Have the fetch request sort the data instead of fetching and then sorting. Fetch requests can have sort descriptors, and the sorting is done by SQLite. This will be far more efficient in both CPU time and memory use.
• Try to avoid fetching everything at once, because with thousands of records that means a lot of memory, no matter what else you do.
• Don't copy the data unless you have some strongly compelling reason to do so. You can use managed objects directly, it's almost never appropriate to duplicate the data like this.
• Since you appear to be using a collection view, consider using NSFetchedResultsController. It's designed to work with table and collection views and will help keep memory and CPU use down.