Can't write large file (> 200MB, 700MB in my case) in ios device?

Question

I am having the problem of writing large file (>200 MB) in iOS device (iPad) but in the simulator it works perfect.
I am using NSFileManager to create file and NSFileData to write file. I think there is no problem in my code as it runs fine in the simulator. Does anyone having the same problem?

To elaborate my situation:

I am saving chunk of files (3MB each) in my device which works fine. That means for a 300 MB file I have 100 chunks. Now, from the 100 chunks I want to create the actual file. So I am using the NSFileManager to create file in first iteration and then using NSFileData to write the 3MB data at the end of the file. While running the program it crashes after 61 chunks. I am guessing there might have some memory related issues in the iPad.

I am saving the chunk of files in fileDir in the format data-0, data-1, data-2... I am applying the decrypt operation on data but for simplicity I have removed that portion.

// List of chunk files        
NSArray *filelist= [[NSFileManager defaultManager] contentsOfDirectoryAtPath:fileDir error:err];
for(int i = 0; i < [filelist count]; i++) {
                // Read the chunk of file
                fileName = [[NSString alloc] initWithFormat:@"data-%d", i];
                filePath = [fileDir stringByAppendingPathComponent:fileName];
                fileReadHandle = [NSFileHandle fileHandleForReadingAtPath:filePath];

                // Write in tempFile
                if(offset == 0){
                    if([[NSFileManager defaultManager] createFileAtPath:tempFile contents:data attributes:nil]){
                        fileWriteHandle = [NSFileHandle fileHandleForWritingAtPath:tempFile];
                        NSLog(@"File was created!");
                    } else {
                        NSLog(@"File was not created.");
                    }
                } else {
                   [fileWriteHandle seekToEndOfFile];  // Tried with comment out this line but same problem
// Write the decrypted data from chunk
            [fileWriteHandle writeData:[[fileReadHandle readDataToEndOfFile] decryptedAES256DataUsingKey:AESEncryptionKey error:err]];
                    }
                }

Edit (11.02.2013) I tried with my previous code where I omitted the data decryption part. Interestingly, the problem was in the decryption part I guess cause without the decryption it works fine. I have added the decryption code. For decryption I am using NSData+CommonCrypto library (it's non ARC) but my project is in ARC.

Answer 1

Modifying the following code worked like a magic,

@autoreleasepool { 
[fileWriteHandle writeData:[[fileReadHandle readDataToEndOfFile] decryptedAES256DataUsingKey:AESEncryptionKey error:err]]; 
}

Answer 2

It could be an operating system issue because the NSFileHandle is never being closed for each chunk. I would recommend closing it.

Also, it looks like you have your variables declared outside the scope of the for loop. Unless you need those variables outside the loop, it's generally good to keep the scope of your variables as small as possible, especially if you are using ARC and are trying to think about when memory will be released.

If you think that the NSFileHandle is holding onto data in memory, try to use the -synchronizeFile method after writing each chunk to make sure in memory changes are reflected to disk.

Also, I moved the creation of the file you are writing to outside the loop, because it's easier to follow for me.

Try this adjustment:

// List of chunk files        
NSArray *filelist= [[NSFileManager defaultManager] contentsOfDirectoryAtPath:fileDir error:err];

if([[NSFileManager defaultManager] createFileAtPath:tempFile contents:[NSData data] attributes:nil]){
    NSLog(@"File was created!");
} else {
    NSLog(@"File was not created.");
}

NSFileHandle *fileWriteHandle = [NSFileHandle fileHandleForWritingAtPath:tempFile];

for(int i = 0; i < [filelist count]; i++) {
    // Read the chunk of file
    NSString *fileName = [NSString stringWithFormat:@"data-%d", i];
    NSString *filePath = [fileDir stringByAppendingPathComponent:fileName];

    NSFileHandle *fileReadHandle = [NSFileHandle fileHandleForReadingAtPath:filePath];
    NSData *data = [fileReadHandle readDataToEndOfFile];

    // No longer using the file
    [fileReadHandle closeFile];

    // Write in tempFile
    [fileWriteHandle writeData:data];
    [fileWriteHandle synchronizeFile];// Flush any data in memory to disk
}

[fileWriteHandle closeFile];