CODEWITHSUNDEEP
python-3.xdockerfluentd
Markus-Hermann
Markus-Hermann

Reputation: 997

Fluentd tcp contact succeeds via netcat tool. But fails via basic Python 3.7. Why?

So I whipped up a docker-based fluentd TCP log collector.

Following the examples here, https://docs.fluentd.org/input/tcp , led to successfully sending a line from my host Win 10 WSL (Debian) by saying

echo "my_service: 08:03:10 INFO [my_py_file:343]: My valuable log info." | netcat 127.0.0.1 5170

This arrived in fluentd as a nice JSON, as hoped-for. But I want to do it from python 3.7! So:

import socket


def netcat(hn: str, p: int, content: bytes):
    """https://www.instructables.com/id/Netcat-in-Python/"""
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((hn, p))
    sock.sendall(content)
    sock.close()


msg_raw = "my_service: 08:03:10 INFO [my_py_file:343]: My valuable log info."

netcat('127.0.0.1', 5170, bytes(msg_raw, 'utf-8'))

WSL or no: This python script runs through, no exceptions. Also no reaction at all from fluentd, which I cannot explain. Could and would any of you?

In case it is of any consequence: Here is the relevant section from my fluentd.conf.

<source>
  @type tcp
  @label mainstream
  @id pawc_tcp
  tag paws.tcp
  port 5170
  bind 0.0.0.0
  # https://docs.fluentd.org/parser/regexp
  <parse>
    @type regexp
    expression /^(?<service_uuid>[a-zA-Z0-9_-]+): (?<logtime>[^\s]+) (?<loglvl>[^\s]+) \[(?<file>[^\]:]+):(?<line>\d+)\]: (?<msg>.*)$/
    time_key logtime
    time_format %H:%M:%S
    types line:integer
  </parse>
</source>

<label mainstream>
  <match paws.tcp>
    @type file
    @id output_tcp
    path /fluentd/log/tcp.*.log
    symlink_path /fluentd/log/tcp.log
  </match>
</label>

Upvotes: 1

Views: 523

Answers (2)

Markus-Hermann
Markus-Hermann

Reputation: 997

As Alex W states above, a \n is needed for the TCP line being accepted by the fluentd regex I use. I'd like to add a second answer to improve the python code of the original question.

There actually is a readily-implemented logging.handler.SocketHandler class! However, it pickles its outputs, looking at a python log server. Using fluentd this means one has to override the emit function to use it. After that all works fine.

import logging, logging.handlers


class SocketHandlerBytes(logging.handlers.SocketHandler):
    def emit(self, record):
        try:
            msg = bytes(self.format(record) + "\n", 'utf-8')
            self.send(msg)
        except Exception:
            self.handleError(record)


sh = SocketHandlerBytes(host, port)
sh.setFormatter(logger_format_appropriate_for_your_fluentd_tcp_regex)
logging.root.addHandler(sh)

Upvotes: -1

Alex W
Alex W

Reputation: 38253

Try sending a \r\n or \0 at the end of your message. The message is being sent as bytes over the network so it's probably being stored in buffers and the code reading the buffer needs a way to know the message is over. The regex is also matching on line terminators so will be necessary I think there as well.

Upvotes: 2

Related Questions