CODEWITHSUNDEEP
c++linuxioofstream
Nasreddin
Nasreddin

Reputation: 1667

How to monitor IO of a program

This open source scientific application written in c++ tends to generate lots of small writes in a short period of time (using ofstream). I am modifying the code such that the IO will not be as frequent.

It really only opens/writes to one or two files. What is the best way to monitor the IO frequency while the program is running? Or maybe to monitor the write frequency of certain files?

Edits: After I changed all the "endl" to "\n", strace shows the program is doing a lot of "writev" instead "write" with original code. Here is the strace output comparison.

Before:

write(4, "+\n", 2)                      = 2
write(4, ">>=@ABCEDBCFBEACDDCDDCBBCCDDCFAC"..., 77) = 77
write(3, "@C3A7DACXX140502:1:1314:6663:801"..., 37) = 37
write(3, "CATCAAACAAGATATGGATGTAGAACGCTCAG"..., 77) = 77
write(3, "+\n", 2)                      = 2
write(3, "?@>CCCDEDDFBDBCECCCEBAECCD@DDEDE"..., 77) = 77
write(4, "@C3A7DACXX140502:1:1314:6663:801"..., 37) = 37
write(4, "TTAGACAATGTAGTGAAATAGTTACTTTGGGG"..., 77) = 77
write(4, "+\n", 2)                      = 2
write(4, "==:BABBCDFBCEBECABBBE@B?DCDBDDCC"..., 77) = 77
write(3, "@C3A7DACXX140502:1:1314:6664:692"..., 36) = 36
write(3, "CAAAGACAGGTATGGAGACAGGAGAAGGGAGC"..., 77) = 77
write(3, "+\n", 2)                      = 2
write(3, ";=;3=6.?A'7*:;@AA=@@?C>.++68>),)"..., 77) = 77
write(4, "@C3A7DACXX140502:1:1314:6664:692"..., 36) = 36
write(4, "ACAAGATGTCTTCGGAGTTTCCGGGATAGCCA"..., 77) = 77

After:

writev(3, [{"\n@C3A7DACXX140502:3:1209:20544:1"..., 8186}, {"ACCTCCTCCTGCTTTCACCTATCCCGCTTCAC"..., 76}], 2) = 8262
writev(4, [{"\n@C3A7DACXX140502:3:1209:20544:1"..., 8186}, {"GATCCTCGTCAGTCCTGAAGGAGTGTCAGCTT"..., 76}], 2) = 8262
writev(3, [{"\n+\nBA@CDEEEDEFEDBBDCDBDBCDCB@DDB"..., 8148}, {"??>ABEEECADDBBABBDCEBDCDCBECDBC@"..., 76}], 2) = 8224
writev(4, [{"\n+\n>>;ABCD@BEDFCDCDECCECCE?DADCE"..., 8148}, {">>;ABABBDDECEDECECCCECBBECCDDDFD"..., 76}], 2) = 8224
writev(3, [{"\n@C3A7DACXX140502:3:1209:20618:6"..., 8184}, {"TTGGAAGGCCAGGTCCAGTAACCGGCCCCATT"..., 76}], 2) = 8260
writev(4, [{"\n@C3A7DACXX140502:3:1209:20618:6"..., 8184}, {"ATTAGTAATTTCAGTGCCTCCTCCATCTTTAG"..., 76}], 2) = 8260
writev(3, [{"\n+\n?@>CCDFEEDDDCBDBDEBBCDB@CDBBB"..., 8148}, {"B@<DDDDFCEDEBBDDBDBDC@EBBECDDCEC"..., 76}], 2) = 8224
writev(4, [{"\n+\n>=;?C@?CBCCEDEAEDCDDBDDBDCDEB"..., 8148}, {">><A@BED@DDBDECBCBECCECDBDCDBEED"..., 76}], 2) = 8224
writev(3, [{"\n@C3A7DACXX140502:3:1209:20684:3"..., 8180}, {"ACCCAAATGAGATCTGTGTGCCAATGTCAGTG"..., 76}], 2) = 8256
writev(4, [{"\n@C3A7DACXX140502:3:1209:20684:3"..., 8180}, {"TCATCTGTGAACTCCACCAAGTTTTGTGCCTC"..., 76}], 2) = 8256

Does that mean it is writing to buffer rather than flushing files?

Upvotes: 0

Views: 1455

Answers (1)

Andrew Henle
Andrew Henle

Reputation: 1

IO tuning and performance improvement is difficult and time-consuming - you have to figure out your application's IO pattern(s) and match both code and hardware to that. And maybe even tune your file system, and you might even go so far as to select a file system based on your IO requirements in the first place. It's kind of like impedance matching electrical components for best performance - everything all the way from application code down to the disk heads themselves matter when you want to drive a system at its design limits.

And you can't drive a system at its design limits while simultaneously ignoring the design because of software abstractions like the C++ IO layer...

  1. Profile your application to find out if IO performance is the bottleneck. If it's not bottlenecked on IO, improving IO won't help application performance much if at all.

  2. Run the application under strace and see what's going on under the hood. Is the application doing a lot of seek()/lseek() system calls? Those invalidate any buffering done by the C++ runtime library and that would call for a revamped IO approach.

  3. If you do find your application needs to improve it's IO performance, you may need to redo the IO without using C++ IO routines - they're not fast. See here: std::fstream buffering vs manual buffering (why 10x gain with manual buffering)?.

  4. If you're just reading/writing large streams without seeking at all and don't re-read the data before it gets flushed out of the cache, the kernel page cache is only slowing you down. Use lower-level C-based IO routines, use page-aligned memory to read/write your data, and use the O_DIRECT flag on opening to perform direct IO and bypass the kernel page cache.

Upvotes: 2

Related Questions