CODEWITHSUNDEEP
c++multithreadingopenmp
Vipul Joshi
Vipul Joshi

Reputation: 48

Is it possible: Using multiple threads to stream output to different files /

I am working on a C++ project to track particles through some known force fields. The code generates huge amount of data in forms of particle's positions and momentum. I am already using openmp directives in the particle tracking routines.However, the overall performance is finally decided by the time taken to write output files. I understand that using threads to write data to a output file is not recommended (and i have tried that). I am curious if there is any way to use multiple threads to write data to multiple files (say I have 4 threads and each thread writes to 4 files at the same time). Can you please suggest me how to proceed? Any tips on how to effectively stream data to files? Thanks in advance

Upvotes: 0

Views: 1094

Answers (1)

JeremiahB
JeremiahB

Reputation: 916

Here is an interesting article on the topic of concurrent I/O: http://www.drdobbs.com/parallel/multithreaded-file-io/220300055

There are a few basic roadblocks that limit the gains for this approach. Number one is hardware capability. HDDs and SSDs have limited read and write speeds, and trying to read/write multiple files simultaneously may not yield a substantial increase in speed. In fact on a hard disk trying to do multiple things at once can actually hurt performance in many situations, which can be seen in the benchmarks at the link I provided. It seems that with 2-4 threads a noticeable gain can be achieved in reading, but the specs on writing were disappointing. Multi threading will definitely help speed up serialization of data though.

Upvotes: 1

Related Questions