CODEWITHSUNDEEP
iosobjective-cmacosconcurrencynsfilemanager
hpique
hpique

Reputation: 120344

Concurrent file enumeration

I have to perform a complex operation on a large number of files. Fortunately, enumeration order is not important and the jobs can be done in parallel without locking.

Does the platform provide a way to do this? For lack of a better API, I was thinking of:

dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
    NSArray *paths = [[NSFileManager defaultManager] subpathsAtPath:folder];
    [paths enumerateObjectsWithOptions:NSEnumerationConcurrent 
    usingBlock:^(NSString *path, NSUInteger idx, BOOL *stop) {
        // Complex operation
    }];
}];

Is there a better way?

Upvotes: 1

Views: 170

Answers (2)

Arek Holko
Arek Holko

Reputation: 9006

Here's an approach you can consider. If you have (or may have) tens of thousands of files, instead of enumerating with enumerateObjectsWithOptions:usingBlock: you may want to enumerate the array manually in batches (let's say 100 elements each). When the current batch completes execution (you can use dispatch groups to check that) you start the next batch. With this approach you can avoid adding tens of thousands of blocks to the queue.

BTW I've deleted my previous answer, because it was wrong.

Upvotes: 1

Wain
Wain

Reputation: 119031

Your current code puts one block on the global queue. So, that single block will run on a background thread and do all of the iteration and processing.

You want to do something a bit different to have your processing tasks run concurrently. You should really do the iteration on the main thread and add a block to the global queue on each iteration of the loop.

Better, create an NSOperation subclass. Put your logic there. Create an instance of the operation in the loop and add them to an operation queue. This is a higher level API and offers you options in adding dependencies, tailoring the maximum concurrency, checking the number of operations still to be completed, etc ...

Upvotes: 1

Related Questions