CODEWITHSUNDEEP
haproxyfluentdfluent-bit
David Bensoussan
David Bensoussan

Reputation: 3205

How to read unescaped json in fluentd?

I have a setup with Fluent Bit sending data in Elasticsearch format to Haproxy in SSL. Haproxy terminates the SSL and forwards the data to Fluentd. Now here is the issue, Fluentd receives the data unescaped and thus can't forward it to ES.

Fluentd receives this data (I added the line break for readability on Stackoverflow):

2020-09-14 11:07:16 +0000 [error]: #0 failed to process request error_class=RuntimeError 
error="Received event is not json: {\"index\":{\"_index\":\"fluent_bit\",\"_type\":\"my_type
\"}}\n{\"@timestamp\":\"2020-09-14T11:07:15.173Z\",\"cpu_p\":3.583333333333333,\"user_p\":2.75,
\"system_p\":0.8333333333333334,\"cpu0.p_cpu\":4,\"cpu0.p_user\":1,\"cpu0.p_system
\":3,\"cpu1.p_cpu\":2,\"cpu1.p_user\":1,\"cpu1.p_system\":1,\"cpu2.p_cpu\":4,\"cpu2.p_user
\":3,\"cpu2.p_system\":1,\"cpu3.p_cpu\":6,\"cpu3.p_user\":4,\"cpu3.p_system\":2,\"cpu4.p_cpu
\":3,\"cpu4.p_user\":3,\"cpu4.p_system\":0,\"cpu5.p_cpu\":6,\"cpu5.p_user\":6,\"cpu5.p_system
\":0,\"cpu6.p_cpu\":4,\"cpu6.p_user\":3,\"cpu6.p_system\":1,\"cpu7.p_cpu\":4,\"cpu7.p_user
\":4,\"cpu7.p_system\":0,\"cpu8.p_cpu\":3,\"cpu8.p_user\":2,\"cpu8.p_system\":1,\"cpu9.p_cpu
\":3,\"cpu9.p_user\":3,\"cpu9.p_system\":0,\"cpu10.p_cpu\":1,\"cpu10.p_user\":0,\"cpu10.p_system
\":1,\"cpu11.p_cpu\":2,\"cpu11.p_user\":2,\"cpu11.p_system\":0}\n"

Multiple notes to add:

  1. I could send from Fluentbit everything in HTTP and it would work, but in this case I lose the timestamp, index and index type.
  2. There must be a parser or filter that simply take the current unescaped json in Fluentd and transform it but I can't find amy in practice. I'm open to any solution, on any stack.

Fluent Bit settings:

[OUTPUT]
    Name             es
    Match            *
    Host             <my-domain>
    Port             443
    Index            fluent_bit
    Type             my_type
    # + TLS settings

Fluentd Settings:

<source>
  @type http
  port 8888
  bind 0.0.0.0
  body_size_limit 32m
  keepalive_timeout 10s
  add_remote_addr true
  format json
</source>

Basic HAProxy backend settings:

backend nodes
  mode          http
  option        forwardfor
  timeout       server 15m
  balance       roundrobin
  server        elastic-us-east-1a ip:port check inter 5000 downinter 500

Upvotes: 0

Views: 524

Answers (1)

Max Lobur
Max Lobur

Reputation: 6040

The reason of this behavior is that you are using [OUTPUT] Name es but sending to a fluentd server instead of ES. If you want logs to go thru centralized log forwarder (fleuntd server) before getting to ES, use this:

FluentBit

...
[OUTPUT]
    Name          forward
...

Fluentd

...
<source>
  type forward
  bind 0.0.0.0
  port 24224
</source>

<match **>
 @type elasticsearch
 ...
</match>
...

FluentBit docs on log forwarding: https://docs.fluentbit.io/manual/pipeline/outputs/forward

OR there's an option to send from the FluentBit to ES directly: https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch

Upvotes: 1

Related Questions