Filebeat container does not send logs to Elastic

Question

On my local machine running Ubuntu 18.04 via "Windows Subsystem Linux 2" on Windows 10, I am running Elastic 7.3, Kibana 7.3 and Elastic 7.3 docker containers.

Set-up is successful and Filebeat seems to monitor containers correctly. However, Kibana does not show any logs.

Setup

To set-up Elastic and Kibana I use the following commands

    docker pull docker.elastic.co/elasticsearch/elasticsearch:7.3.1
    docker run --network=lognetwork --name=elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.3.1

    docker pull docker.elastic.co/kibana/kibana:7.3.1
    docker run --name=kibana --network=lognetwork -e ELASTICSEARCH_HOSTS=http://elasticsearch:9200 -p 5601:5601 docker.elastic.co/kibana/kibana:7.3.1

After these two commands, the logs from Kibana container show it successfully connects to Elastic:

    {"type":"log","@timestamp":"2019-09-01T13:22:18Z","tags":["status","plugin:spaces@7.3.1","info"],"pid":6,"state":"green","message":"Status changed from yellow to green - Ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

I can also go to Kibana dashboard on http://localhost:5601 as well as Elastic on http://localhost:9200 both function properly

I then set up filebeat:

    docker run --network=lognetwork docker.elastic.co/beats/filebeat:7.3.1 setup -E setup.kibana.host=kibana:5601 -E output.elasticsearch.hosts=["elasticsearch:9200"]

I can see both Elastic and Kibana container logs and returning 200. The logs on the Filebeat container show:

    Index setup finished.
    Loading dashboards (Kibana must be running and reachable)
    Loaded dashboards
    Loaded machine learning job configurations
    Loaded Ingest pipelines

Finally, I pull the default config from Elastic site, launch Filebeat and attach to the container

    curl -L -O https://raw.githubusercontent.com/elastic/beats/7.3/deploy/docker/filebeat.docker.yml

    docker run -d --network=lognetwork  --name=filebeat   --user=root   --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro"   --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro"   --volume="/var/run/docker.sock:/var/run/docker.sock:ro"   docker.elastic.co/beats/filebeat:7.3.1 filebeat -e -strict.perms=false   -E output.elasticsearch.hosts=["elasticsearch:9200"]
    docker attach filebeat

I can see Filebeat sending monitoring pulse but when it does, elastic logs do not show anything new.

To test, I launch Docker "hello-world" which generates several lines of logs

    docker run hello-world

Filebeat shows the following log

    2019-09-01T13:30:40.624Z        INFO    log/input.go:148        Configured paths: [/var/lib/docker/containers/460cc8c215ff69ecf28685c9cf89c0e56d0b3e4f680b8bf29beb5b570ebb7a14/*-json.log]
    2019-09-01T13:30:40.624Z        INFO    input/input.go:114      Starting input of type: container; ID: 16402101064670842079

I then go to http://localhost:5601

Results:
Kibana shows no logs. Clicking for "check for new data" does not show anything either.

The folder /var/lib/docker/containers is also empty. The path returned by filebeat log (/var/lib/docker/containers/460cc8c215ff69ecf28685c9cf89c0e56d0b3e4f680b8bf29beb5b570ebb7a14/) does not seem to exist.

Expected:
- Kibana to show the "hello world" docker container logs
- To see a log file under /var/lib/docker/containers

What am I missing?

Thank you,
Olivier

Answer 1

In my case :

Docker desktop installed in Windows 10 + WSL2 enabled in docker.

• I was trying to use file beat to collect logs of all docker containers.
• ELK + Filebeat were also running as docker containers.

The pipeline: Filebeat -> logstash -> elastic search -> kibana

Problem: Filebeat was not finding logs from docker. But from a local mounted folder it was sending logs to ELK and was showing up in kibana.

Solution: I was running docker-compose up from wsl bash shell. Instead I ran the same from windows powershell, or cmd and the logs from docker containers started to appear in kibana.

In docker-compose file:

filebeat:
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - /var/lib/docker:/var/lib/docker
        - ./MYLOG_TEST:/usr/share/filebeat/mylog
        - ./MY_filebeat.yml:/usr/share/filebeat/filebeat.yml

and in MY_filebeat.yml:

filebeat.inputs:
  #for docker logs
  - type: container # for older filestream version use docker as type
    enabled: true
    paths:
       - /var/lib/docker/containers/**/*.log

  #for my test log files
  - type: log # for filebeat latest versions8.1+, use filestream as type
    enabled: true
    paths:
      - /usr/share/filebeat/mylog/*.log

Answer 2

Well, it took me many hours before asking on SO, and of course, 30mn after asking I found the answer.

The trick was to check where the logs were created as running Docker-Desktop on WSL2 is slightly different than running Docker on Linux.

docker inspect filebeat | grep LogPath

returns:

"LogPath": "/var/data/docker-desktop/default/daemon-data/containers/fd56c5e43c9206baaadd33d3a711e523107622450d0deafb498e7940d809f779/fd56c5e43c9206baaadd33d3a711e523107622450d0deafb498e7940d809f779-json.log

Then changing the volume map accordingly volume="/var/data/docker-desktop/default/daemon-data/containers:/var/lib/docker/containers:ro" when launching filebeat did the job:

docker run -d 
    --network=lognetwork  
    --name=filebeat   
    --user=root   
    --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro"   
    --volume="/var/data/docker-desktop/default/daemon-data/containers:/var/lib/docker/containers:ro"   
    --volume="/var/run/docker.sock:/var/run/docker.sock:ro" 
    docker.elastic.co/beats/filebeat:7.3.1 filebeat -e -strict.perms=false -E output.elasticsearch.hosts=["elasticsearch:9200"]

The logs are now properly shown on kibana