Python's SyslogHandler and TCP

Question

I'm trying to understand why the SyslogHandler class from Python's logging framework (logging.handlers) does not implement any of the framing mechanism described by RFC 6587:

1. Octet Counting: it "prepends" the message length to the syslog frame:

2. Non-Transparent-Framing: a trailer character to separate messages. This is what most of the servers understand.

This "problem" can be easily solved by adding a LF character to the end of the messages, however I would expect that the SyslogHandler would take care of this by default:

sHandler = logging.handlers.SysLogHandler(address=(address[0], address[1]), socktype = socket.SOCK_STREAM)
sHandler.setFormatter(logging.Formatter(fmt=MSG_SYSLOG_FORMAT, datefmt=DATE_FMT))
self.addHandler(sHandler)

This does not work neither with Fluentd, nor with rsyslog. As I said, I've temporarily added this to line 855 of handlers.py (just to test):

msg = prio + msg + '\n'

And now is working.

---

My questions:

1. Should the Python SyslogHandler class offer the possibility to set on/off the octet counting or trailer character. Currently it does nothing...
2. It is the job of the programmer to know how the server works and override the Handler to address the message framing?

For now, what I'm doing now is to override emit() method, sub-classing SyslogHandler.

Answer 1

Thanks to @Martijn Pieters♦ for His answer, my answer expands his answer.

I implemented a class that inherited from the SyslogHandler class and override the emit function. I also opened a pull request for this issue: https://github.com/python/cpython/pull/24556

python2:

import socket
import logging.handlers as handlers


class TcpSyslogHandler(handlers.SysLogHandler):
    """
    This class override the python SyslogHandler emit function.
    It is needed to deal with appending of the nul character to the end of the message when using TCP.
    Please see: https://stackoverflow.com/questions/40041697/pythons-sysloghandler-and-tcp/40152493#40152493
    """
    def __init__(self, message_separator_character, address=('localhost', handlers.SYSLOG_UDP_PORT),
                 facility=handlers.SysLogHandler.LOG_USER,
                 socktype=None):
        """
        The user of this class must specify the value for the messages separator.
        :param message_separator_character: The value to separate between messages.
                                            The recommended value is the "nul character": "\000".
        :param address: Same as in the super class.
        :param facility: Same as in the super class.
        :param socktype: Same as in the super class.
        """
        super(SfTcpSyslogHandler, self).__init__(address=address, facility=facility, socktype=socktype)

        self.message_separator_character = message_separator_character

    def emit(self, record):
        """
        SFTCP addition:
        To let the user to choose which message_separator_character to use, we override the emit function.
        ####
        Emit a record.

        The record is formatted, and then sent to the syslog server. If
        exception information is present, it is NOT sent to the server.
        """
        try:
            msg = self.format(record) + self.message_separator_character

            """
            We need to convert record level to lowercase, maybe this will
            change in the future.
            """
            prio = '<%d>' % self.encodePriority(self.facility, self.mapPriority(record.levelname))
            # Message is a string. Convert to bytes as required by RFC 5424
            if type(msg) is unicode:
                msg = msg.encode('utf-8')
            msg = prio + msg
            if self.unixsocket:
                try:
                    self.socket.send(msg)
                except socket.error:
                    self.socket.close()  # See issue 17981
                    self._connect_unixsocket(self.address)
                    self.socket.send(msg)
            elif self.socktype == socket.SOCK_DGRAM:
                self.socket.sendto(msg, self.address)
            else:
                self.socket.sendall(msg)
        except (KeyboardInterrupt, SystemExit):
            raise
        except Exception:
            self.handleError(record)

python3:

import socket
import logging.handlers as handlers


class SfTcpSyslogHandler(handlers.SysLogHandler):
    """
    This class override the python SyslogHandler emit function.
    It is needed to deal with appending of the nul character to the end of the message when using TCP.
    Please see: https://stackoverflow.com/questions/40041697/pythons-sysloghandler-and-tcp/40152493#40152493
    """
    def __init__(self, message_separator_character, address=('localhost', handlers.SYSLOG_UDP_PORT),
                 facility=handlers.SysLogHandler.LOG_USER,
                 socktype=None):
        """
        The user of this class must specify the value for the messages separator.
        :param message_separator_character: The value to separate between messages.
                                            The recommended value is the "nul character": "\000".
        :param address: Same as in the super class.
        :param facility: Same as in the super class.
        :param socktype: Same as in the super class.
        """
        super(SfTcpSyslogHandler, self).__init__(address=address, facility=facility, socktype=socktype)

        self.message_separator_character = message_separator_character

    def emit(self, record):
        """
        SFTCP addition:
        To let the user to choose which message_separator_character to use, we override the emit function.
        ####
        Emit a record.

        The record is formatted, and then sent to the syslog server. If
        exception information is present, it is NOT sent to the server.
        """
        try:
            msg = self.format(record) + self.message_separator_character
            if self.ident:
                msg = self.ident + msg

            # We need to convert record level to lowercase, maybe this will
            # change in the future.
            prio = '<%d>' % self.encodePriority(self.facility,
                                                self.mapPriority(record.levelname))
            prio = prio.encode('utf-8')
            # Message is a string. Convert to bytes as required by RFC 5424
            msg = msg.encode('utf-8')
            msg = prio + msg
            if self.unixsocket:
                try:
                    self.socket.send(msg)
                except OSError:
                    self.socket.close()
                    self._connect_unixsocket(self.address)
                    self.socket.send(msg)
            elif self.socktype == socket.SOCK_DGRAM:
                self.socket.sendto(msg, self.address)
            else:
                self.socket.sendall(msg)
        except Exception:
            self.handleError(record)

Answer 2

Given that the question is tagged fluentd, have you tried using fluent.handler.FluentHandler in place of logging.handlers.SysLogHandler - see https://github.com/fluent/fluent-logger-python?

Answer 3

Syslog support in logging predates the RFC, and before that RFC, there was little in the way of standards.

To be precise: the SysLogHandler handler was part of logging when first added to the Python standard library in 2002 and has remained largely the same since (TCP support was added in 2009, and RFC5424 support was improved in 2011); the original code was based on this syslog module from 1997.

From other bug reports it is clear the maintainers want to keep the broadest backwards compatibility in code here, so if you need specific functionality from a newer RFC, you have two options:

• Extend the class and implement that functionality yourself
• Submit feature requests and / or patches to improve the functionality in the logging module; take into account the backwards-compatibility requirements.