CODEWITHSUNDEEP
multithreadingloggingpoco-libraries
DaManJ
DaManJ

Reputation: 430

Poco logging - possible to buffer log entries?

I am logging market data with Poco asynchronously using FileChannel and AsyncChannel, which is creating huge amounts of log entries per second. It appears that Poco writes every single message separately to file and does not buffer. I believe this is putting considerable strain on my HDD/filesystem, and I had an application crash which I believe is related.

Is there anyway to have Poco only save the log to disk in say 1Mb increments, and also write anything remaining in the buffer to file on close of the logger?

And separately, is there any chance this is going to create huge numbers of threads? From what i've read the AsyncChannel just puts messages into a queue, so I guess only 1 additional thread is created?

Following is basically the code I am using:

#include "Poco/Message.h"
#include "Poco/FormattingChannel.h"
#include "Poco/PatternFormatter.h"
#include "Poco/Logger.h"
#include "Poco/FileChannel.h"
#include "Poco/AutoPtr.h"
#include "Poco/AsyncChannel.h"

class APocoClass
{
private:
    Poco::AutoPtr<Poco::FileChannel> pFileChannel;
    Poco::AutoPtr<Poco::PatternFormatter> pPF;
    Poco::AutoPtr<Poco::FormattingChannel> pFormattingChannel;
    Poco::AutoPtr<Poco::AsyncChannel> pFileChannelAsync;
    Poco::Logger & _pocoLogger;
public:

    APocoClass() :
        pFileChannel(new Poco::FileChannel()),
        pPF(new Poco::PatternFormatter("%Y%m%d %H:%M:%S.%F: %t")),
        pFormattingChannel(new Poco::FormattingChannel(pPF, pFileChannel)),
        pFileChannelAsync(new Poco::AsyncChannel(pFormattingChannel)),
        _pocoLogger(Poco::Logger::create("PocoLogger", pFileChannelAsync, Poco::Message::PRIO_INFORMATION))
    {

        pFileChannelAsync->setProperty("priority", "lowest");
        pFileChannel->setProperty("path", "MsgStorage/poco.log");
        pFileChannel->setProperty("rotation", "daily");
        pFileChannel->setProperty("times", "utc");
        pFileChannel->setProperty("archive", "timestamp");

    }

    ~APocoClass() {
        _pocoLogger.shutdown();
        _pocoLogger.close();
        pFileChannelAsync = nullptr;
        pFileChannel = nullptr;
    }

    //following is called every time we have a new market data message to log
    void MessageReceived(const string & message) {
        Poco::Message m("PocoLogger", message, Poco::Message::Priority::PRIO_INFORMATION);
        _pocoLogger.log(m);
    }

}

Upvotes: 1

Views: 1249

Answers (1)

Alex
Alex

Reputation: 5330

Is there anyway to have Poco only save the log to disk in say 1Mb increments, and also write anything remaining in the buffer to file on close of the logger?

You don't have such precise level of control through FileChannel, but you have available flush property (default true) which determines whether buffers are flushed on every log entry. Set it to false and see if things improve.

If that does not satisfy your performance requirements, then you have an option of writing your own wrapper, see LogStream for an example - obviously, you'll want to implement your own logic for LogStreamBuf::writeToDevice(). (It would be much simpler if library allowed you to simply pass in your own streambuf, but unfortunately it does not.)

And separately, is there any chance this is going to create huge numbers of threads?

No, AsyncChannel will launch itself in a thread and process all notifications (ie. log messages) in that single thread.

Upvotes: 4

Related Questions