CODEWITHSUNDEEP
grafana-lokipromtail
Allan Xu
Allan Xu

Reputation: 9318

Promtail syslog job is not showing any error, but can’t see syslog logs in queries

I use the docker image grafana/promtail:2.9.2 to deploy my promtail. The varlog jobs are working well.

I use this configuration to have a syslog receiver:


server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: syslog-receiver
  syslog:
    listen_address: 0.0.0.0:40514
    idle_timeout: 12h
    use_incoming_timestamp: true
    labels:
      job: syslog-receiver

  relabel_configs:
    - source_labels: ['__syslog_message_hostname']
      target_label: host
    - source_labels: ['__syslog_message_severity']
      target_label: level
    - source_labels: ['__syslog_message_facility']
      target_label: syslog_facility
    - source_labels: ['__syslog_message_app_name']
      target_label: syslog_identifier

I can see varlogs log entries, working well.

I use the following Python app to test syslogs:

import logging
import logging.handlers

my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.DEBUG)

handler = logging.handlers.SysLogHandler(address = ('v145',40514))

my_logger.addHandler(handler)

my_logger.debug('this is debug')
my_logger.critical('this is critical')
It sends logs to my promtail container. No error. I checked docker logs, no error.

However, this query does not return any logs:

`{job="syslog-receiver"} |= ```

How can I diagnose this issue?

Upvotes: 0

Views: 335

Answers (1)

Allan Xu
Allan Xu

Reputation: 9318

I painfully found the answer:

Promtail syslog is VERY limited. The team suggest to add a syslog-ng in front of promtail. Means Loki <- Promtail <- syslog-ng

https://grafana.com/blog/2021/03/23/how-i-fell-in-love-with-logs-thanks-to-grafana-loki/

The alternate approach is to ise syslog-ng 's loki destination, and eliminate promtail.

Upvotes: 0

Related Questions