Promtail + Loki - Only shows some namespaces not all

Question

we recently decided to install loki and promtail via the loki-stack helm chart. Loki and promtail kind of work. We get some logs from Promtail and we can visualize them in grafana but our development namespace is nowhere to be found in loki. Promtail shows the development pod as an active target and promtail already collected the logs from the pod but we cant seem to get them into loki somehow... Any ideas?

Answer 1

I managed to resolve this by adding Promtail as a daemonSet.

In the ClusterRoleBinding section at the very bottom, you need to set subjects.namespace to the namespace that Promtail is deployed in. My namespace differed because I was deploying with ArgoCD that added it to the monitoring namespace.

Here is the full daemonSet config:

--- # Daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail-daemonset
spec:
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      serviceAccount: promtail-serviceaccount
      containers:
      - name: promtail-container
        image: grafana/promtail
        args:
        - -config.file=/etc/promtail/promtail.yaml
        env: 
        - name: 'HOSTNAME' # needed when using kubernetes_sd_configs
          valueFrom:
            fieldRef:
              fieldPath: 'spec.nodeName'
        volumeMounts:
        - name: logs
          mountPath: /var/log
        - name: promtail-config
          mountPath: /etc/promtail
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
      volumes:
      - name: logs
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: promtail-config
        configMap:
          name: promtail-config
--- # configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
    - url: https://{YOUR_LOKI_ENDPOINT}/loki/api/v1/push

    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: pod-logs
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - docker: {}
      relabel_configs:
        - source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          replacement: $1
          separator: /
          source_labels:
            - __meta_kubernetes_namespace
            - __meta_kubernetes_pod_name
          target_label: job
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: pod
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
            - __meta_kubernetes_pod_uid
            - __meta_kubernetes_pod_container_name
          target_label: __path__

--- # Clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail-clusterrole
rules:
  - apiGroups: [""]
    resources:
    - nodes
    - services
    - pods
    verbs:
    - get
    - watch
    - list

--- # ServiceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail-serviceaccount

--- # Rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promtail-clusterrolebinding
subjects:
    - kind: ServiceAccount
      name: promtail-serviceaccount
      namespace: your-namespace-here
roleRef:
    kind: ClusterRole
    name: promtail-clusterrole
    apiGroup: rbac.authorization.k8s.io

Answer 2

I faced with similar problem, I was not seeing traefik namespace.

The problem was, that promtail was missing tolerations so it could run on nodes on which traefik pods were running.

Answer 3

tl;dr

set loki.monitoring.selfMonitoring.grafanaAgent.installOperator to false

---

This problem is caused by grafana-agent which is installed by default as a sub-chart of grafana/loki chart...

agent creates secret 'loki-logs-config' (loki in this case is Helm release name) which contains following configuration:

agent.yml: |+
  logs:
      configs:
          - clients:
              - external_labels:
                  cluster: loki
                url: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push
            name: monitoring/loki
            scrape_configs:
              - job_name: podLogs/monitoring/loki
                kubernetes_sd_configs:
                  - namespaces:
                      names:
                          - monitoring
                    role: pod
                pipeline_stages:
                  - cri: {}
                relabel_configs:
                  - source_labels:
                      - job
                    target_label: __tmp_prometheus_job_name
                  - action: keep
                    regex: loki
                    source_labels:
                      - __meta_kubernetes_pod_label_app_kubernetes_io_instance
                  - action: keep
                    regex: loki
                    source_labels:
                      - __meta_kubernetes_pod_label_app_kubernetes_io_name
                  - source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - source_labels:
                      - __meta_kubernetes_service_name
                    target_label: service
                  - source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: monitoring/loki
                    target_label: job
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: monitoring/$1
                    source_labels:
                      - __meta_kubernetes_pod_controller_name
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - action: replace
                    replacement: loki
                    target_label: cluster
      positions_directory: /var/lib/grafana-agent/data
  server: {}

As you can see under kubernetes_sd_configs there is namespaces list with value of monitoring - I have no idea why is it there, but that's the namespace I've installed this chart into. You won't see this secret after executing helm template - it seems that Grafana Agent creates it somehow after startup. It has label app.kubernetes.io/managed-by=grafana-agent-operator
Pretty magical if you ask me...

The solution for me was disabling disabling installation of Grafana Agent:

loki:
  loki:
    commonConfig:
      replication_factor: 1
    storage:
      type: 'filesystem'
    auth_enabled: false
  monitoring:
    dashboards:
      enabled: false
    selfMonitoring:
      enabled: true
      grafanaAgent:
        installOperator: false
      lokiCanary:
        enabled: false

Note: top loki element in the code block above is needed only if you add grafana/loki chart as subchart to your chart

---

IMO enabling beta feature (Grafana Agent is v0.30.0 today) in a Chart used as a reference in Loki's doc is insane :)