Match Grafana timestamp with the one in the log entry

Question

I have Promtail running on one container, scraping the logs from a file. It parses them inside a pipeline, and sends them to Loki. The promtail.yaml relevant part is:

      - targets:
          - localhost
        labels:
          environment: staging
          vm: 4
          operator: blabla
          app_name: blabla_app
          instance: 5
          job: app.error.log
          __path__: /var/log/blabla_app/error.log
    pipeline_stages:
      - match:
          selector: '{job=~"app.error.log"}'
          stages:
          - json:
              expressions:
                time: timestamp
                level: level
                log:
          - labels:
              level:
          - timestamp:
              format: RFC3339Nano
              source: time

However, in case of Promtail being stopped/crashes, it will report a bunch of logs at the same time, in which case Loki will send multiple log lines in Grafana. When that happens, Grafana will sort the logs not by the timestamp inside the log line, but by the time the logs were sent, which ultimatelly floods my log dashboards with false information.

How can I make it so when I query the logs, they are selected based on the timestamp in them, rather than the timestamp that either Loki or Grafana puts when it receives them?

Grafana screenshot of the visual representation of the logs:

I have tried changing the format a few times, tried to reject old samples from Loki, which could help as a workaround, but doesn't provide a resolution.

I am expecting the outcome where when I select the last 5 minutes in Grafana, the logs with timestamps in them in the last 5 minutes to show, not the ones that have been reported in the last 5 minutes.

Answer 1

The issue was with incorrect timestamp on the promtail pipeline stage.

Instead of getting the Unix timestamp form the log entry (which could have possibly worked), I decided to use 'time' as the timestamp, and modified the yaml as follows:

          - json:
              expressions:
                time: time
                level: level
                log:
          - labels:
              level:
          - timestamp:
              format: 2006-01-02 15:04:05.000
              source: time

After that it started picking it up successfully.