CODEWITHSUNDEEP
dockerdocker-composefluentd
Genzer
Genzer

Reputation: 2991

Docker does not send logs to fluentd when it runs in the same Docker Compose stack

I'm trying to use Fluentd in Docker Compose stack base on the tutorial https://docs.fluentd.org/container-deployment/docker-compose.

My stack is as follow:

version: '3'

services:
  one:
    image: alpine:3.10.2
    command: sh -c 'while true; do echo $$(hostname) - $$(date); sleep 2; done'
    depends_on:
      - fluentd
    logging:
      driver: "fluentd"
      options:
        fluentd-async-connect: 'true'
        fluentd-address: localhost:24224
        tag: service.one

  two:
    image: alpine:3.10.2
    command: sh -c 'while true; do echo $$(hostname) - $$(date); sleep 1; done'
    depends_on:
      - fluentd
    logging:
      driver: "fluentd"
      options:
        fluentd-async-connect: 'true'
        fluentd-address: localhost:24224
        tag: service.two

  fluentd:
    image: fluent/fluentd:v1.7-debian-1
    volumes:
      - ./fluentd/conf:/fluentd/etc
      - ./logs:/apps/logs
    ports:
      - "24224:24224"
      - "24224:24224/udp"

My fluent.conf is pretty simple:

<source>
    @type forward
    port 24224
    bind 0.0.0.0
</source>

<match **>
    @type stdout
</match>

<match *.*>
    @type file
    path /apps/logs/service.one/${tag}.%Y_%m_%d__%H-%M-%S.log
    # time_slice_format %Y%m%d
    # time_format %Y_%m_%d__%H-%M-%S

    <buffer tag,time>
        timekey 1d
        timekey_use_utc true
        flush_interval 30s
    </buffer>
</match>

I'm using

# Docker for Mac
$ docker -v                                 
Docker version 19.03.1, build 74b1e89
$ docker-compose -v 
docker-compose version 1.24.1, build 4667896b

When I start the stack with docker-compose up -d, the fluentd container does not receive any log from the other containers.

I discovered that if I started the fluentd first and then the others, the logging will work as expected. (Or --force-recreate the other containers work the same).

Does anyone encounter the same issue? Or fluentd is supposed to be used that way?

Thanks a lot for your time!

Upvotes: 3

Views: 4914

Answers (2)

crazygit
crazygit

Reputation: 489

@Genzer

I meet the same issue as yours, It seems like the fluentd service MUST BE READY before other service run.

You can control the serivces startup order by @LinPy's link

https://docs.docker.com/compose/startup-order/

Upvotes: 0

Stefano
Stefano

Reputation: 5076

I think the problem is that you've got 2 match directives in the configuration and the first one blocks the second.

Try with this one:

<source>
    @type forward
    port 24224
    bind 0.0.0.0
</source>


<match *.*>
    @type copy
    <store>
        @type file
        path /apps/logs/temp/log.${tag}.txt
        # time_slice_format %Y%m%d
        # time_format %Y_%m_%d__%H-%M-%S

        <buffer tag, time>
            @type file
            path /apps/logs/${tag}/entry_%Y%m%d.log
            timekey 1m
            timekey_wait 30s
            timekey_use_utc true
        </buffer>
    </store>
    <store>
        @type stdout
    </store>
</match>

Upvotes: 2

Related Questions