Why does `<< std::endl` not call the operator I want it to call?

Question

I was looking for a solution to write to a file and the console at the same time. I found a nice solution here.

As I am working pre C++11 I had to make a small change to the code from Lightness Races in Orbit:

#include <iostream>
#include <fstream>
#include <string>

struct OutputAndConsole : std::ofstream
{
    OutputAndConsole(const std::string& fileName)
       : std::ofstream(fileName.c_str())   // constructor taking a string is C++11
       , fileName(fileName)
    {};

    const std::string fileName;
};

template <typename T>
OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var)
{
    std::cout << var;    
    static_cast<std::ofstream&>(strm) << var;
    return strm;
};

It works nicely apart from a small thing takes puzzles me. If I use it like this:

int main(){
    OutputAndConsole oac("testLog.dat");
    double x = 5.0;
    oac << std::endl;
    static_cast<OutputAndConsole&>(oac << "foo \n" << x << "foo").operator<<(std::endl);  
    oac << "foo" << std::endl;
}

then all the std::endl are ignored for the output on the console while they appear correctly in the file. My guess is that when I use std::endl the ostream::operator<< is called which will print to the file but not to the console. The line with the static_cast<OutputAndConsole&> is my dilettantish attempt to call the correct operator, but still only the line break from \n appears on the console.

Why for std::endl the wrong operator is called?

How can I call the correct one?

PS: I know that I can use \n without problems, but still I would like to know what is going on here and how to fix it.

Answer 1

I had a similar problem, and fixed it by making my operator<< function a friend like this:

struct OutputAndConsole : std::ofstream
{
    OutputAndConsole(const std::string& fileName)
       : std::ofstream(fileName.c_str())   // constructor taking a string is C++11
       , fileName(fileName)
    {};

    const std::string fileName;

    template <typename T>
    friend OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var)
    {
        std::cout << var;    
        std::ofstream::operator << (var); //*See note at end
        return strm;
    }
};

* This is the preferred syntax for calling a base class method/operator from a derived class method/operator. While your syntax will work in most cases, it would fail if std::ofstream operator << were declared pure-virtual in ofstream.

Answer 2

Let us try something simpler:

#include <iostream>

struct Foo { };

template <typename T>
Foo& operator<<(Foo& foo, const T& var)
{
    std::cout << var;
    return foo;
};

int main(){
    Foo foo;
    foo << std::endl;
}

This does not compile:

a1.cpp: In function ‘int main()’:
a1.cpp:14:9: error: no match for ‘operator<<’ (operand types are ‘Foo’ and ‘<unresolved overloaded function type>’)
     foo << std::endl;
         ^
a1.cpp:14:9: note: candidates are:
a1.cpp:6:6: note: template<class T> Foo& operator<<(Foo&, const T&)
 Foo& operator<<(Foo& foo, const T& var)
      ^
a1.cpp:6:6: note:   template argument deduction/substitution failed:
a1.cpp:14:17: note:   couldn't deduce template parameter ‘T’

Why? What does the compiler try to tell us?

The definition of std::endl can be found here: http://en.cppreference.com/w/cpp/io/manip/endl

template< class CharT, class Traits >
std::basic_ostream<CharT, Traits>& endl( std::basic_ostream<CharT, Traits>& os );

So std::endl is not a variable. It is a template. More precisely, a template function. In my little code, the compiler is unable to instantiate the template.

When we directly call std::cout << std::endl; , the compiler instantiates std::endl from CharT and Traits of decltype(std::cout) .

In your code, the compiler instead instantiates the template using CharT and Traits from std::ofstream, because your OutputAndConsole is a descendant of std::ofstream. When std::cout tries to output the wrong instantiation of std::endl, it fails.

PS: The last paragraph is only partially correct. When you write

oac << something;

where something has type T,

It can, theoretically, call either of two

std::ofstream& std::ofstream::operator<<(T)  // or operator<<(const T&)
// -- or --
OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var);

The first definition is possible because OutputAndConsole inherited the member function operator<< from std::ofstream. The second form is provided by you.

When something is a variable, it uses the second definition.

When something is a template, it cannot use the second definition, as there is no way to determine the parameters of the template. So it uses the first definition. Therefore,

oac << std::endl;  // std::endl is a template

is equivalent to

static_cast<ofstream&>(oac) << std::endl;

We can see it by the following code:

#include <iostream>

struct Foo : std::ofstream {};

template <typename T>
Foo& operator<<(Foo& strm, const T& var)
{
    std::cout << "X" << std::endl;
    return strm;
};

int main() {
    Foo oac;
    oac << std::endl;
}

This code does NOT print "X".

Answer 3

struct OutputAndConsole : std::ofstream
{
  // ...
};

template <typename T>
OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var);

As others have mentioned, std::endl is a template function. A template function is not a value, it is just a name.

A template function can be converted to a value if you try to pass it to a function expecting function of a compatible signature. It is not converted to a value if passed to a template function taking T or const T&, because the name of a template function represents a whole host of possible values.

Because std::endl is not a valid argument for your custom-written operator<<, it looks elsewhere. It finds the std::ofstream's operator<< that takes an io manipulator function by explicit function pointer.

That one works, it can convert endl to that function pointer type! So, gleefully, it calls it.

To fix this problem, add an operator<< overload to OutputAndConsole that takes io manipulator function pointers.

The easiest way to do that is to write a helper function:

template <class T>
void output_to(OutputAndConsole& strm, const T& var)
{
  std::cout << var;    
  static_cast<std::ofstream&>(strm) << var;
};

then two << overloads:

template<class T>
OutputAndConsole& operator<<(OutputAndConsole& strm, const T& var) {
  output_to(strm, var);
  return strm;
}
OutputAndConsole& operator<<(OutputAndConsole& strm, std::ostream& (*var)(std::ostream&)) {
  output_to(strm, var);
  return strm;
}

which causes the std::endl template to find a matching <<.

Answer 4

I suggest not to implement standard I/O stream functionality via the stream-interface, but the streambuffer-interface. A customized stream-interface leads usually to trouble as soon as the stream is recognized as a std::istream/std::ostream, only (due to some operator, manipulator or function taking a reference to a stream).

You may utilize:

#include <array>
#include <iostream>
#include <sstream>
#include <vector>

// BasicMultiStreamBuffer
// ============================================================================

/// A (string) stream buffer for synchronizing writes into multiple attached buffers.
template<class Char, class Traits = std::char_traits<Char>, class Allocator = std::allocator<Char> >
class BasicMultiStreamBuffer : public std::basic_stringbuf<Char, Traits, Allocator>
{
    // Types
    // =====

    private:
    typedef typename std::basic_stringbuf<Char, Traits> Base;

    public:
    typedef typename std::basic_streambuf<Char, Traits> buffer_type;
    typedef typename buffer_type::char_type char_type;
    typedef typename buffer_type::traits_type traits_type;
    typedef typename buffer_type::int_type int_type;
    typedef typename buffer_type::pos_type pos_type;
    typedef typename buffer_type::off_type off_type;

    private:
    typedef typename std::vector<buffer_type*> container_type;

    public:
    typedef typename container_type::size_type size_type;
    typedef typename container_type::value_type value_type;
    typedef typename container_type::reference reference;
    typedef typename container_type::const_reference const_reference;
    typedef typename container_type::iterator iterator;
    typedef typename container_type::const_iterator const_iterator;


    // Construction/Destructiion
    // =========================

    public:
    BasicMultiStreamBuffer()
    {}

    template <typename...Buffers>
    BasicMultiStreamBuffer(Buffers* ...buffers) {
        std::array<buffer_type*, sizeof...(Buffers)> buffer_array{buffers...};
        m_buffers.reserve(buffer_array.size());
        for(auto b : buffer_array) {
            if(b)
                m_buffers.push_back(b);
        }
    }

    template <typename Iterator>
    BasicMultiStreamBuffer(Iterator first, Iterator last)
    :   m_buffers(first, last)
    {}

    ~BasicMultiStreamBuffer() {
        sync();
    }


    private:
    BasicMultiStreamBuffer(BasicMultiStreamBuffer const&); // No Copy.
    BasicMultiStreamBuffer& operator=(BasicMultiStreamBuffer const&); // No Copy.


    // Capacity
    // ========

    public:
    bool empty() const { return m_buffers.empty(); }
    size_type size() const { return m_buffers.size(); }


    // Iterator
    // ========

    public:
    iterator begin() { return m_buffers.begin(); }
    const_iterator begin() const { return m_buffers.end(); }
    iterator end() { return m_buffers.end(); }
    const_iterator end() const { return m_buffers.end(); }


    // Modifiers
    // =========

    public:
    /// Attach a buffer.
    void insert(buffer_type* buffer) {
        if(buffer) m_buffers.push_back(buffer);
    }

    /// Synchronize and detach a buffer.
    void erase(buffer_type* buffer) {
        iterator pos = this->begin();
        for( ; pos != this->end(); ++pos) {
            if(*pos == buffer) {
                char_type* p = this->pbase();
                std::streamsize n = this->pptr() - p;
                if(n)
                    sync_buffer(*pos, p, n);
                m_buffers.erase(pos);
                break;
            }
        }
    }


    // Synchronization
    // ===============

    private:
    int sync_buffer(buffer_type* buffer, char_type* p, std::streamsize n) {
        int result = 0;
        std::streamoff offset = 0;
        while(offset < n) {
            int k = buffer->sputn(p + offset, n - offset);
            if(0 <= k) offset += k;
            else {
                result = -1;
                break;
            }
            if(buffer->pubsync() == -1)
                result = -1;
        }
        return result;
    }

    protected:
    /// Synchronize with the attached buffers.
    /// \ATTENTION If an attached buffer fails to synchronize, it gets detached.
    virtual int sync() override {
        int result = 0;
        if( ! m_buffers.empty()) {
            char_type* p = this->pbase();
            std::streamsize n = this->pptr() - p;
            if(n) {
                iterator pos = m_buffers.begin();
                while(pos != m_buffers.end()) {
                    if(0 <= sync_buffer(*pos, p, n)) ++pos;
                    else {
                        pos = m_buffers.erase(pos);
                        result = -1;
                    }
                }
            }
        }
        this->setp(this->pbase(), this->epptr());
        if(Base::sync() == -1)
            result = -1;
        return result;
    }

    private:
    container_type m_buffers;
};

typedef BasicMultiStreamBuffer<char> OStreamBuffers;


// BasicMultiStream
// ============================================================================

template<class Char, class Traits = std::char_traits<Char>, class Allocator = std::allocator<Char> >
class BasicMultiStream : public std::basic_ostream<Char, Traits>
{
    // Types
    // =====

    private:
    typedef std::basic_ostream<Char, Traits> Base;

    public:
    typedef BasicMultiStreamBuffer<Char, Traits, Allocator> multi_buffer;
    typedef std::basic_ostream<Char, Traits> stream_type;

    typedef typename multi_buffer::buffer_type buffer_type;
    typedef typename multi_buffer::char_type char_type;
    typedef typename multi_buffer::traits_type traits_type;
    typedef typename multi_buffer::int_type int_type;
    typedef typename multi_buffer::pos_type pos_type;
    typedef typename multi_buffer::off_type off_type;

    typedef typename multi_buffer::size_type size_type;
    typedef typename multi_buffer::value_type value_type;
    typedef typename multi_buffer::reference reference;
    typedef typename multi_buffer::const_reference const_reference;
    typedef typename multi_buffer::iterator iterator;
    typedef typename multi_buffer::const_iterator const_iterator;


    // Construction
    // ============

    public:
    BasicMultiStream()
    :   Base(&m_buffer)
    {}

    template <typename ...Streams>
    BasicMultiStream(Streams& ...streams)
    :   Base(&m_buffer), m_buffer(streams.rdbuf()...)
    {}

    private:
    BasicMultiStream(const BasicMultiStream&); // No copy.
    const BasicMultiStream& operator = (const BasicMultiStream&); // No copy.

    // Capacity
    // ========

    public:
    bool empty() const { return m_buffer.empty(); }
    size_type size() const { return m_buffer.size(); }


    // Iterator
    // ========

    public:
    iterator begin() { return m_buffer.begin(); }
    const_iterator begin() const { return m_buffer.end(); }
    iterator end() { return m_buffer.end(); }
    const_iterator end() const { return m_buffer.end(); }


    // Modifiers
    // =========

    public:
    template <typename StreamIterator>
    void insert(StreamIterator& first, StreamIterator& last)
    {
        while(first != last)
            insert(*first++);
    }
    void insert(stream_type& stream) { m_buffer.insert(stream.rdbuf()); }
    void erase(stream_type& stream) { m_buffer.erase(stream.rdbuf()); }

    private:
    multi_buffer m_buffer;
};

typedef BasicMultiStream<char> MultiStream;


int main() {
    MultiStream s(std::cout, std::cerr, std::clog);
    s << "Hello World" << std::endl;
    printf("[Three lines of output]\n");
}

Note, the only function applying changes to the std:: basic_streambuf interface is virtual int sync() override.

The standard basic stream classes do not provide any interface besides deriving and initializing a custom stream class. The actual (virtual) interface is the standard stream buffer basic_streambuf.

Answer 5

To make it work, I would create my own set of manipulators:

struct ManipEndl {};
const ManipEndl manip_endl;
OutputAndConsole& operator<<(OutputAndConsole& strm, const ManipEndl& foo)
{
    std::cout << std::endl;    
    static_cast<std::ofstream&>(strm) << '\n'; // no need for std::endl here
    return strm;
};

int main(){
    OutputAndConsole oac("testLog.dat");
    double x = 5.0;
    oac << manip_endl;
    oac << x << manip_endl << "foo" << manip_endl;  
    oac << "foo" << manip_endl;
}

Answer 6

std::endl is a function, not a string. Your overloaded method takes a string for the overloading, so it isnt this one who get called when you do << std::endl

You need to create an operator who takes a function who has the same signature as std:endl to do your overload.

 std::ostream& operator<<( std::ostream& (*f)(std::ostream&) )