CODEWITHSUNDEEP
loggingtimestampgrafana-loki
Sunbro _
Sunbro _

Reputation: 19

Why Loki shows "timestamp too new" when i try to parse logs?

I've tried to parse custom logs. Loki not accept parsed timestamps cause:

level=error ts=2020-09-18T07:17:52.655628313Z msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry for stream has timestamp too new: 2020-09-18 10:01:52.935 +0000 UTC"

But old logs with time, for example: 2020-09-09 10:01:52.935 or even 2020-09-17 10:01:52.935 loki parsing normally

My log:

2020-09-18 10:01:52.935;message

Steps to reproduce the behavior: grafana/loki:latest grafana/promtail:latest

My promtail config:

...
    scrape_configs:
  - job_name: myjob
    pipeline_stages:
    - regex:
        expression: '^(?P<timestamp>[\d-\s:.]{20,30});(?P<message>.*)'
    - timestamp:
        source: timestamp
        format: 2006-01-02 15:04:05.999
    static_configs:
    - targets:
        - localhost
      labels:
        job: myjob
        _path_: /logs/*log

My loki config:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 792h

storage_config:
  boltdb:
    directory: /tmp/loki/index

  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: false

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: true
  retention_period: 792h

Upvotes: 0

Views: 4864

Answers (1)

sach3000
sach3000

Reputation: 31

In promtail pipeline stage try this format:

pipeline_stages:
    - regex:
        expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2}\S\d{2}:\d{2}:\d{2}.\d+\+\d{2}:\d{2}).*'
    - timestamp:
        source: timestamp
        format: RFC3339Nano
        action_on_failure: skip

Upvotes: 1

Related Questions