Promtail Config for Docker Swarm

Question

I am currently trying to set up log monitoring for a docker swarm cluster using promtail, loki and grafana. The forwarding of the logs from promtail to loki and the visualisation in graphana is all working fine.

However, with my current promtail configuration all container logs get send unagregated to loki. Therefore my question would be, if anybody is aware of a promtail configuration, which sends the container logs aggregated by the docker swarm service they are belonging to ?

The current promtail config.yml looks like following:

- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs_prom
      __path__: /var/log/*log

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs_prom
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        output: log
        stream: stream
        attrs:
  - json:
      expressions:
        tag:
      source: attrs
  - regex:
      expression: (?P<image_name>(?:[^|]*[^|])).(?P<container_name>(?:[^|]*[^|])
).(?P<image_id>(?:[^|]*[^|])).(?P<container_id>(?:[^|]*[^|]))
      source: tag
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      tag_prom:
      stream_prom:
      image_name_prom:
      container_name_prom:
      image_id_prom:
      container_id_prom:
  - output:
      source: output

Thanks in advance !!

Answer 1

I find out you can use Docker driver instead of Promtail. You can easily install it with:

$ docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

And verify installation with:

docker plugin ls

Then you can configure to use loki log driver for all containers with default logging configuration:

{
    "debug" : true,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "https://<user_id>:<password>@logs-us-west1.grafana.net/loki/api/v1/push",
        "loki-batch-size": "400"
    }
}

From documentation, I am not sure if logs are also kept on localhost in this case but if you just set logging driver manually into docker-compose.yml file, it will be sent into Loki and also kept on localhost. From documentation:

version: "3.7"
services:
  logger:
    image: grafana/grafana
    logging:
      driver: loki
      options:
        loki-url: "https://<user_id>:<password>@logs-prod-us-central1.grafana.net/loki/api/v1/push"

This configuration also keep stack and service name:

stack name and service name for each swarm service and project name and service name for each compose service are automatically discovered and sent as Loki labels, this way you can filter by them in Grafana.

I hope this helps.

EDIT1: I find similar question on GrafanaLabs comunity: Add container name to promtail docker logs

EDIT2: From Known issues:

The driver keeps all logs in memory and will drop log entries if Loki is not reachable and if the quantity of max_retries has been exceeded. To avoid the dropping of log entries, setting max_retries to zero allows unlimited retries; the drive will continue trying forever until Loki is again reachable. Trying forever may have undesired consequences, because the Docker daemon will wait for the Loki driver to process all logs of a container, until the container is removed. Thus, the Docker daemon might wait forever if the container is stuck.

The result is, if docker loki driver cannot connect to the loki, it will be stucked an not printing any logs to stdout of docker container. It will be also unable to stop container, so it could cause problem with deployment of the application. As mentioned in doc, it is recommended to use docker target or even better Docker service discover. It is able to get metadata from docker like container name, id, network, labels, etc... To use it you have to use replabel config, example from doc:

scrape_configs:
  - job_name: flog_scrape 
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: name
            values: [flog] 
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

Be careful with this configuration, label container, even with one app, can contain a lot of uniq values because of docker hash name like:

• nginx_nginx.1.0zwo879s92fk38o07a65uzolc - replicated deployment
• minio_minio.wy6c6pk1arod6vbqeem8iyosj.t9mdy2hgnv2upqwmrh0fo2c9p - global deployment

This is not recommended according to Grafana Loki label best pracitce. I fix this with regexp relabel config which filter name of the container, in above example nginx_nginx or minio_minio, and replica number (if exists):

  relabel_configs:
    - source_labels: ['__meta_docker_container_name']
      regex: '/(.*)\.[0-9]\..*'
      target_label: 'name'
    - source_labels: ['__meta_docker_container_name']
      regex: '/(.*)\.[0-9a-z]*\..*'
      target_label: 'name'
    - source_labels: ['__meta_docker_container_name']
      regex: '/.*\.([0-9]{1,2})\..*'
      target_label: 'replica'