CODEWITHSUNDEEP
prometheus
ThinkGeek
ThinkGeek

Reputation: 5147

Prometheus error - Error on ingesting samples

I am getting following error in Prometheus on monitoring my Node JS based web application.

level=warn ts=2019-08-02T18:23:48.658364708Z caller=scrape.go:932 component="scrape manager" scrape_pool=batch_web target=https://example.com:443/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=6

Any pointers on why this could be happening and how can I solve this?

Upvotes: 8

Views: 26647

Answers (4)

mati kepa
mati kepa

Reputation: 3231

I had the very same issue. The thing was that the global config for CloudWatch Explorer looked like this:

period_seconds: 300
range_seconds: 3600

after changing it to below and setting range_seconds as default , problem yelld

period_seconds: 120

Upvotes: 0

Arjun Pandey- parjun8840
Arjun Pandey- parjun8840

Reputation: 56

I had the similar issue ( on CentOS7 OS). In my case the /etc/ntp.conf was pointing to a non-existing server, resulting in out-of-sync server time.

In the prometheus server logs I was getting below error message-

target=http://localhost:9090/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=380

Solution:

You can use the below public servers in your /etc/ntp.conf file:

server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst

Follow then below steps:

  1. Restart ntp service

    systemctl ntpd restart

  2. Stop Prometheus service

    systemctl stop prometheus

Go to the Prometheus data directory and remove everything inside it:

ls -lrt
drwxr-xr-x 2 prometheus prometheus     6 Oct 20  2021 chunks_head
-rw-r--r-- 1 prometheus prometheus     0 Oct 19 21:18 lock
-rw-r--r-- 1 prometheus prometheus 20001 Oct 19 21:20 queries.active
drwxr-xr-x 2 prometheus prometheus    66 Oct 19 21:18 wal
  1. Start Prometheus service

Upvotes: 1

030
030

Reputation: 11719

  • Enable debugging mode in Prometheus by passing --log.level=debug as an argument to prometheus

  • Check the log. In my case the metric that was dropped, was returned:

    msg="Out of order sample" series="some-metric{a="b",c="d"}

  • Check the prometheus configuration and rule files that could return a duplicated some-metric. In my case a duplicated rule was causing the issue as the same metric was scraped from the federated prometheus and calculated again by my prometheus. Removing the duplicated rule solved the issue.

Upvotes: 3

jpietras
jpietras

Reputation: 1

Working with prometheus on virtual machines in VMware, vagrant and Paralles virtual environments, all running Ubuntu 18.04, I had the same promblem with this message in syslog

msg="Error on ingesting out-of-order samples"
and veraious num_dropped= values

The thing that worked was to issue ALL of these command, on each image, then reboot the image. Without the reboot the num_dropped began to be lower but the error persisted until the reboot

/usr/bin/timedatectl set-ntp false
systemctl stop systemd-timesyncd
systemctl disable systemd-timesyncd
systemctl mask systemd-timesyncd

Upvotes: 0

Related Questions