CODEWITHSUNDEEP
jsondockerkubernetesloggingfluent
Jason Nanay
Jason Nanay

Reputation: 115

FluentD cannot parse the log file content

I'm trying to collect logs using FluentD and I'm getting following error;

2021-08-31 05:37:03 +0000 [warn]: #0 [tail_container_logs] pattern not match: "2021-08-31T05:37:03.133785104Z stderr F I0831 05:37:03.130864       1 main.go:227] handling current node"
2021-08-31 05:37:06 +0000 [warn]: #0 [tail_container_logs] pattern not match: "2021-08-31T05:37:05.474234466Z stdout F Hello world"
2021-08-31 05:37:08 +0000 [warn]: #0 [tail_container_logs] pattern not match: "2021-08-31T05:37:07.195447969Z stderr F 2021-08-31 05:37:07.195301 I | etcdserver/api/etcdhttp: /health OK (status code 200)"

My fluent.conf file is as follows;

<source>
  @type tail
  @id tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/fluent/log/containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag kubernetes.*
  exclude_path "/var/log/containers/my-fluent*.log"
  read_from_head true
  <parse>
    @type json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

<match kubernetes.**>
  @type file
  @id   output1
  path         /var/fluent/log/log/data.*.log
  append       true
</match>

When I check the Docker logging driver configuration on the host (not on kubernetes nodes) where the kind kubernetes cluster runs, it shows json-file as the logging driver.

However I couldn't find the logging driver configuration in the node for CRI. When I connected to the node and ran the crictl info it shows the configuration but there was no logging driver. After further investigation I found cri-o logging has no json support. I suspect only option available is to use @type regex in FluentD configuration.

Any suggestion on how to parse the cri-o logs in fluentD?

Thanks in advance!!!

Upvotes: 1

Views: 1420

Answers (1)

Jason Nanay
Jason Nanay

Reputation: 115

I was able to get the cri-o logs parsed using regex found in fluent-bit topic. Here is the fluentD configuration of the part.

<source>
  @type tail
  @id tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/fluent/log/containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag kubernetes.*
  exclude_path "/var/log/containers/my-fluent*.log"
  #format json_in_json
  read_from_head true
  <parse>
    @type regexp
    expression /^(?<time>.+)\b(?<stream>stdout|stderr)\b(?<log>.*)$/
    time_key time
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </parse>
</source>

Currently there is no cri-o parser for FluentD

Upvotes: 1

Related Questions