CODEWITHSUNDEEP
dockernetwork-programmingdocker-composefluentdfluent-bit
naraghi
naraghi

Reputation: 518

How do I make fluent-bit forward docker logs?

I am trying to run a fluent-bit container according to https://www.velebit.ai/blog/tech-blog-collecting-logs-in-docker-clusters/. Note the ports section in the docker-compose.yml. I have td-agent running (fluentd) on port 24224 on localhost, so I can use the fluentd logging driver for docker. The use case is that I want to forward docker logs with fluent-bit. Here is my /etc/docker/daemon.json:

{
   "log-driver": "fluentd",
   "log-opts": {
     "fluentd-address": "localhost:24224"
   }
}

fluent-bit.conf:

[SERVICE]
  flush                 1
  log_level             info
  parsers_file          parsers_multiline.conf

[INPUT]
  name                  tail
  refresh_interval      5
  path                  /var/log/*.log
  read_from_head        true

[INPUT]
  Name          forward
  Listen        0.0.0.0 
  port          24224

[FILTER]
  name                  multiline
  match                 *
  multiline.key_content log
  multiline.parser      go, multiline-regex-test

[OUTPUT]
  Name http
  Match *
  Host data-prepper
  Port 2021
  URI /log/ingest
  Format json

When I do docker-compose up, I get ERROR: for fluent-bit Cannot start service fluent-bit: driver failed programming external connectivity on endpoint fluent-bit (0f3884b5ca7302aa58356c28bfb041ed413554f0b9608d6476720126545779dc): Error starting userland proxy: listen tcp4 0.0.0.0:24224: bind: address already in use

How do I actually make this work?

Upvotes: 0

Views: 3478

Answers (1)

Sebastian
Sebastian

Reputation: 11

I would look at what exactly is occupying port 24224 on your host:

ss-tunlp | grep 24224

enter image description here

In my case, this port is occupied by the docker-proxy process, which actually translates packets into the fluent-bit container.

If you have the same, then most likely you started and then did not kill the fluent-bit container. You can check like this:

docker ps

enter image description here

If this is the case, and if it's not production, you can simply terminate the old container using its CONTAINER ID:

docker stop 3d3a9074f145

Upvotes: 0

Related Questions