How to make C++ endl manipulator thread safe?

Question

I have a multithreaded program in C++. The problem I face while trying to print something in the log through multiple threads and program crashes. The specific problem is that I have cout << "some log message" << endl; And when I see the pstack for the dumped core, it shows that endl caused the problem of contention. On one thread, I have:

 ff308edc _IO_do_write (ff341f28, ff341f6f, 2, ff341f6f, fc532a00, ff141f74) + dc
 ff3094d8 _IO_file_overflow (ff341f28, a, ff000000, 1c00, 0, fffc00) + 2a8
 ff3101fc overflow__7filebufi (ff341f28, a, 0, 1ffee, 7f2082, ff1b4f18) + 8
 ff314010 overflow__8stdiobufi (a, a, ff314000, 4, fc532a00, fbdfbd51) + 10
 ff306dd4 __overflow (ff341f28, a, 4, ff1b5434, ff1b5784, 82c8c) + 20
 ff30fdd0 _IO_putc (a, ff341f28, 7d5be4, ff314048, ff1b5784, 82c8c) + 34
 ff313088 endl__FR7ostream (7d5be0, 20, fbdfbd4e, 1, 0, 76f) + c
 ff32a3f8 __ls__7ostreamPFR7ostream_R7ostream (7d5be0, 3bfb74, 3bf800, 385cd8, 76f, 0) + 4

On another thread, I have:

 --- called from signal handler with signal 11 (SIGSEGV) ---
 ff312f20 flush__7ostream (7d5be0, a, 4, ff1b5434, ff1b5784, 82c8c) + 10
 ff312f58 flush__FR7ostream (7d5be0, ff341f28, 7d5be4, ff314048, ff1b5784, 82c8c) + 4
 ff313090 endl__FR7ostream (7d5be0, 20, fbffbd4e, 1, 0, 232a) + 14

std::cout is buffered, and std::endl forces a flush of the output stream. So, it seems that on one thread endl is doing the flushing of the buffer while the other thread is trying to putc the newline characeter and hits an overflow.

Possible solutions (but have issues) could be: (1) Have an independent thread safe logger class that can be used for all log output, so instead of using std::cout we may use logger::cout in all places -- that's tedious as logging is scattered all over the place. also, to make this thread safe, mutex lock and unlock needs to be before and after each attempt to call insertion operator << or manipularors like endl. That's a performance hit. (2) Instead of using endl, we may use '\n' instead, so that flushing is not forced on every new line insertion, rather flushed when need be, by the underlying ostream buffering mechanism. But, is this thread safe? Not sure. (3) Switch to C++11 as C++11's std::cout is supposed to thread safe. But that's not immediately possible.

Any other better alternative or thoughts to get rid of SIGSEGV causing from endl manipulator by concurrent threads?

Can I somehow fore a synchronization / mutual exclusion while calling endl?

Answer 1

If you're using C++11, any access to a common object from more than one thread must be protected. There is an exception if none of the accesses mutate the object, and there is a special exception for the standard iostream objects (but not streams in general), but even then, the standard clearly says that the individual characters may be interleaved, so this exception really doesn't buy you anything; it will prevent the core dump, but won't prevent the output from being gibberish, so you need some sort of synchronization even then.

In pre-C++11, every implementation had its own rules; some even made each << atomic, in all streams. But given something like:

std::cout << a << b;

, none guaranteed that output from another thread couldn't occur between the output of a and the output of b, so this really didn't buy you anything.

The result is that you do need some sort of thread safe logger class. Generally, such logger classes will collect the data in an instance local "collector". This might be an std::string or an std::vector<char>, embedded into a custom streambuf, which knows about log records, inserts a time stamp at the front, etc., and very importantly, ensures that the full log record is output atomically at the end of the record. I usually manage this by using some sort of forwarding logger class, which is instantiated as a temporary for each log record, and notifies the underlying streambuf (one per thread) each time it is constructed and destructed, so the streambuf can take care of the rest. If you don't need things like timestamps, etc., you can do this somewhat simpler by implementing a streambuf which never outputs to the final destination except on explicit calls to flush. (This does require some discipline on the client side, to ensure that flush is called at appropriate moments. The temporary wrapper solution has the advantage of handling this more or less automatically.)

Finally, except in small throw-away programs, you should never output to std::cout. You either output to some sort of logger object (or a stream obtained from such an object), or you output to an std::ostream& passed as an argument to your function. Setting up output and the actual output are two separate concerns, and will normally be handled at different places in the program. The code doing the output just deals with an std::stream, which it has received from elsewhere.

If you're dealing with a large body of existing code, which was written without this principle being taken into account: you always have the possibility of modifying the output streambuf of std::cout. This will not solve the problem of interleaving, but it can be made to be thread safe otherwise, so at least you won't crash.

Answer 2

I never thought about your problem in much detail, so this is just a quick guess on how I would solve your problem, but it might have major flaws.

Basically, I would write a wrapper class around streams that guards the stream operator and gives a special meaning to SomeManipulator (like std::endl).

template <class T>
struct Wrapper
{
    Wrapper( T& stream );

    template <class U>
    Wrapper& operator<<( const U& u )
    {
        lock if thread does not hold the lock.
        forward u to stream.
    }

    Wrapper& operator<<( SomeManipulator )
    {
        pre-cond: thread holds lock. // I.e., you can't print empty lines.
        forward std::endl to stream.
        unlock.
    }
};

Please note, that this introduces a major overhead for output, depending on your situation, you may want to prefer writing to a separate stream with each thread and combining them later on.

Answer 3

It's not just endl, the whole output stream is shared. It has to be that way really. It's a single common resource. And the library does not know the serialization that you desire. You have to add that in your code.

Just this about what happens if you don't serialize output. Different pieces of output can get all mixed up with each other, even if you somehow manage to avoid runtime errors. So you have to decide the atomic units of output in your program, and serialize them.