Background upload of large amount of data

Question

I need to upload large number of photos to a server from my iOS device. For example, 500 photos. How should I do this right?

I created upload task with background session configuration with NSURLSession for each photo. I tried making my own custom queue where each next task would launch after completion of previous one. But at one moment new task wasn't starting. I guess, because it all was happening in the background. More on this issue, you can read here

(If approach with a queue was right, would you please advise a good realisation of a queue for async tasks, because I may messed up something in my implementation)

So after article linked above, my guess was I should start all upload tasks at once (not one after another). But I have efficiency concern. If I would create a task for each photo, it would 500 background async task and about 1 gigabyte of a data uploading parallel. I guess, it would cause some problem with network.

To sum up all said above, which is the right way to upload large piece of data in background in iOS (500 photos in my case)?

Answer 1

I recently had a similar requirement where I needed to handle the background upload of a large number of media files (75 in my case). I was able to successfully implement it using URLSession with background configuration. The key was managing the session properly, handling progress updates, and ensuring that the upload continued even if the app went into the background. Here's how I approached the problem...

// Helper extension to append Data
extension Data {
mutating func append(_ string: String) {
    if let data = string.data(using: .utf8) {
        append(data)
    }
  }
}

class BackgroundUploader: NSObject, URLSessionDelegate, URLSessionTaskDelegate, URLSessionDataDelegate {
var backgroundSession: URLSession!
var uploadCompletionHandlers: [Int: (String) -> Void] = [:] // Store completion handlers for each task
var files = String()
override init() {
    super.init()
    
    // Create a background configuration
    let config = URLSessionConfiguration.background(withIdentifier: "com.yourapp.upload")
    config.isDiscretionary = false // Ensure uploads run even when the app is in the background
    config.sessionSendsLaunchEvents = true // App will relaunch if terminated during upload
    
    // Create the background session
    backgroundSession = URLSession(configuration: config, delegate: self, delegateQueue: nil)
    NotificationCenter.default.addObserver(self, selector: #selector(appWillEnterForeground), name: UIApplication.willEnterForegroundNotification, object: nil)
}
deinit {
    NotificationCenter.default.removeObserver(self) // Remove observer when this object is deallocated
}
// Function to upload an array of data with headers and completion handler
func uploadDataArray(dataArray: [Data], to url: URL, completionHandler: @escaping (String) -> Void) {
    let boundary = "Boundary-\(UUID().uuidString)" // Generate a unique boundary for the request
    let multipartBody = createMultipartBody(dataArray: dataArray, boundary: boundary)
    
    // Write multipart body to a temporary file
    let tempDirectory = FileManager.default.temporaryDirectory
    let tempFileURL = tempDirectory.appendingPathComponent(UUID().uuidString)
    
    do {
        // Write the multipart body to a temp file
        try multipartBody.write(to: tempFileURL)
        
        // Upload the file
        uploadFile(fileURL: tempFileURL, to: url, boundary: boundary, completionHandler: completionHandler)
        
    } catch {
        print("Failed to write multipart body to temporary file: \(error)")
    }
}
// Create a multipart body with all files from the array
private func createMultipartBody(dataArray: [Data], boundary: String) -> Data {
    var body = Data()
    
    for (index, data) in dataArray.enumerated() {
        // Add each file to the body as "files[]"
        let filename = "file\(index + 1).jpg" // Update the filename for each file
        let mimeType = "image/jpeg" // Change MIME type as needed

        body.append("--\(boundary)\r\n")
        body.append("Content-Disposition: form-data; name=\"files[]\"; filename=\"\(filename)\"\r\n")
        body.append("Content-Type: \(mimeType)\r\n\r\n")
        body.append(data)
        body.append("\r\n")
    }
    
    // Final boundary to signal the end of the multipart form
    body.append("--\(boundary)--\r\n")
    
    return body
}
// Upload the file using file URL with headers
func uploadFile(fileURL: URL, to url: URL, boundary: String, completionHandler: @escaping (String) -> Void) {
    var request = URLRequest(url: url)
    request.httpMethod = "POST"
    
    // Set custom headers (multipart/form-data and Authorization headers)
    request.setValue("multipart/form-data; boundary=\(boundary)", forHTTPHeaderField: "Content-Type")
    request.setValue("Bearer \(AIUser.sharedManager.token)", forHTTPHeaderField: "Authorization")
    request.setValue(APP_VERSION ?? "", forHTTPHeaderField: "app-version")
    
    // Create a background upload task for the file
    let task = backgroundSession.uploadTask(with: request, fromFile: fileURL)
    
    // Store the completion handler associated with the task ID
    uploadCompletionHandlers[task.taskIdentifier] = completionHandler
    
    task.resume()
}
// Handle response and pass uploaded file name to completion handler
func urlSession(_ session: URLSession, dataTask: URLSessionDataTask, didReceive data: Data) {
    do {
        // Parse the JSON response
        if let responseJSON = try JSONSerialization.jsonObject(with: data, options: []) as? [String: Any] {
            print("Response JSON: \(responseJSON)")
            
            // Check if responseData contains images
            if let responseData = responseJSON["responseData"] as? [String: Any],
               let images = responseData["images"] as? [String] {
                
              
                // Combine the image names into a single string (if needed)
                let joinedFileNames = images.joined(separator: ", ")
                self.files = joinedFileNames
                if let completionHandler = uploadCompletionHandlers[dataTask.taskIdentifier] {
                    completionHandler(joinedFileNames) // Pass the uploaded file names back
                }
                
                // Call the associated completion handler with the uploaded file names
                
            } else if let successMessage = responseJSON["message"] as? String {
                // Handle the message if images aren't present
                if let completionHandler = uploadCompletionHandlers[dataTask.taskIdentifier] {
                    completionHandler(successMessage) // Pass the response message back
                }
            }
        }
    } catch {
        print("Failed to parse response: \(error)")
    }
}
// URLSessionTaskDelegate: Handle completion in the background
func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
    if let error = error {
        print("Upload failed with error: \(error.localizedDescription)")
    } else {
        print("Upload completed successfully for task: \(task.taskIdentifier)")
        scheduleUploadCompletionNotification()
    }
    
    // Remove the stored handler for the completed task
    uploadCompletionHandlers.removeValue(forKey: task.taskIdentifier)
}

// Optional: Handle app relaunch after task completion in background
func urlSessionDidFinishEvents(forBackgroundURLSession session: URLSession) {
    // Notify the system that all background events have been handled
    print("All background events have been handled.")
}

@objc func appWillEnterForeground() {
       // Check if there are any active tasks
       backgroundSession.getAllTasks { tasks in
           if tasks.isEmpty {
               // No active tasks, call your API from the controller
               DispatchQueue.main.async {
                   // Assume you have a reference to your controller, call the API here
                   print("Entered in forground and uploaded files are \(self.files)")
                   
               }
           } else {
               print("There are still \(tasks.count) uploads in progress.")
           }
       }
   }
func scheduleUploadCompletionNotification() {
    let content = UNMutableNotificationContent()
    content.title = "Upload Complete"
    content.body = "All your files have been uploaded successfully."
    content.sound = .default

    // Set a trigger to show the notification immediately
    let trigger = UNTimeIntervalNotificationTrigger(timeInterval: 1, repeats: false)

    // Create the notification request
    let request = UNNotificationRequest(identifier: UUID().uuidString, content: content, trigger: trigger)

    // Add the notification request to the system
    let center = UNUserNotificationCenter.current()
    center.add(request) { error in
        if let error = error {
            print("Error scheduling notification: \(error)")
        } else {
            print("Notification scheduled!")
        }
      }
    }
  }

use:-
var uploader : BackgroundUploader!
uploader.uploadDataArray(dataArray: [Data], to: URL, completionHandler: (String) -> Void)

Answer 2

Unfortunately, Apple's APIs are terrible at this task because of bad design decisions. There are a couple of major obstacles that you face:

• There are limits to how many simultaneous tasks you can create. I think performance starts to break down at somewhere on the order of 100 tasks.
• There are limits to how often the OS will wake your app up. The more often it wakes your app, the longer it will wait before waking it again. At some point, this will result in not being able to schedule new tasks.
• Uploads don't continue from where they left off in the event of a failure; they restart. This can result in huge bandwidth costs on a bad network.

I suspect that the best approach is to:

• Chunk the requests into several large groups, each of which can be written to a single ZIP archive that is not so big that the user runs out of disk space, but not so small that it uploads too quickly.
• Write the first set of those files into a single file (e.g. in ZIP format).
• Use a custom script on the server side that lets you resume uploads from where it left off by adding extra CGI parameters.
• On failure, ask the server how much data it got, then truncate the front of the file and reupload from the current spot.
• On success, compute how quickly the first large file finished uploading, and if it is not O(minutes), combine the next few sets. Write the set/sets to a file, and start the next request.

With the caveat that all of this must be done fairly quickly. You may find it necessary to pre-combine the files into ZIP archives ahead of time to avoid getting killed. But do not be tempted to combine them into a single file, because then you'll take too long when truncating the head on retry. (Ostensibly, you could also provide any parameters as part of the URL, and make the POST body be raw data, and provide a file stream to read from the ZIP archive starting at an offset.)

If you're not banging your head against a wall already, you soon will be. :-)

Answer 3

For downloading this much amount of images you may need to ask the user not to stop the application or putting in background mode.

Because in that two cases we won't be able to perform this much large task to get done.

If the user phone is in active state,

Create a NSoperation corresponding to each upload process. In your case it may be around 500.

And add thus NSOperations into a queue called NSOperationQueue and just start the NSOperationQueue tasks.

It will perform one by one.

For more details like caching and all.. please follow the SO POST

Here is the Swift Version