CODEWITHSUNDEEP
cmultithreadingperformancefileoperating-system
user1295872
user1295872

Reputation: 509

Multiple threads on different cores reading same set of files

I have a multi threaded process, where each thread runs on one core. I am reading the same set of files from each of the threads and processing them. Will reading the same set of files by multiple threads affect the performance of the process?

Upvotes: 3

Views: 349

Answers (3)

Alexander Altshuler
Alexander Altshuler

Reputation: 3064

You may consider Memory-Mapped Files for concurrent read access.

It will avoid overhead of copying data into every process address space.

Upvotes: 1

Hugo Fonseca
Hugo Fonseca

Reputation: 67

Reading the same set of files from each different thread may reduce the performance of the process, because the IO request are normally costly and slow, in addition to being repeating the same read operation for each difference thread.

One possible solution to deal with this is having one thread dealing with the IO reads/writes and the rest processing the data, for example as a producer consumer.

Upvotes: 2

NotGaeL
NotGaeL

Reputation: 8484

Not necessarily, but there are a few factors to be taken on account.

When you open a file for READING you don't need to put a read lock on it.

That means multiple threads can be reading from the same file.

In fact all threads from a process share the process memory, so you can use that for your benefit by caching the whole set (or part of it, depending on the size) on the process memory. That will reduce access time.

Otherwise if we assume all files are in the same device, the problem is that reading multiple files simultaneously from the same device at the same time is slow and, depending on the number of threads and the storage type it can be noticeably slower

Upvotes: 3

Related Questions