CODEWITHSUNDEEP
linuxmultithreadingstdout
LiMar
LiMar

Reputation: 2972

Sharing stdout among multiple threads/processes

I have a linux program(the language doesn't matter) which prints it's log onto stdout. The log IS needed for monitoring of the process.

Now I'm going to parallelize it by fork'ing or using threads.

The problem: resulting stdout will contain unreadable mix of unrelated lines...

And finally The question: How would you re-construct the output logic for parallel processes ?

Upvotes: 6

Views: 8697

Answers (4)

LiMar
LiMar

Reputation: 2972

Sorry for answering myself...

The definite solution was to use the GNU parallel utility.

It came to replace the well known xargs utility, but runs the commands in parallel, separating the output into groups.

So I just left my simple one-process, one-thread utility as-is and piped its call through the parallel like that:

generate-argument-list | parallel < options > my-utility

This, depending on parallel's options can produce nicely grouped outputs for multiple calls of my-utility

Upvotes: 3

Brady
Brady

Reputation: 10357

If its multithreaded, then you'll need to mutex protect printing/writing to the stdout log. The most common way to do this in Linux and c/c++ is with pthread_mutex. Additionally, if its c++, boost has synchronization that could be used.

To implement it, you should probably encapsulate all the logging in one function or object, and internally lock and unlock the mutex.

If logging blocking performance becomes prohibitive, you could consider buffering the log messages (in the afore mentioned object or function) and only writing to stdout when the buffer is full. You'll still need mutex protection to buffer, but buffering will be faster than writing to stdout.

If each thread will have its own logging messages, then they will still need to all share the same mutex for writing to stdout. In this case it would probably be best for each thread to buffer its respective log messages and only write to stdout when the the buffer is full, thus only obtaining the mutex for writing to stdout.

Upvotes: 2

tuxuday
tuxuday

Reputation: 3037

Another approach, which we use, is to delegate a thread, logger thread, for logging. All other threads wishing to log will send it to logger thread. This method gives you flexibility as formatting of logs can be done is single place, which can also be configurable. If you don't want to worry about locks can use sockets for message passing.

Upvotes: 3

Richard Harrison
Richard Harrison

Reputation: 19403

If you're in C++ I'd consider using Pantheios or derivative version Boost::Log or using look at Logging In C++ : Part 2 or

If you're in another language then file locking around the IO operations is probably the way to go see File Locks, you can achieve the same results using semaphonres or any other process control system but for me file locks are the easiest.

You could also consider using syslog if this monitoring is considered as system wide.

Upvotes: 2

Related Questions