CODEWITHSUNDEEP
prometheus
sow
sow

Reputation: 59

Prometheus service is failing to start

I am using the Prometheus for past 15 days. from yesterday the service is failing because of out of memory.

I have increased the memory from 4GB to 8GB but now I am unable to start the Prometheus service and failing with below error :

Nov 15 10:25:20 prometheus: level=info ts=2019-11-15T18:25:20.393Z caller=head.go:598 component=tsdb msg="head GC completed" duration=22.375774ms
Nov 15 10:25:35 prometheus: level=info ts=2019-11-15T18:25:35.135Z caller=head.go:668 component=tsdb msg="WAL checkpoint complete" first=549 last=550 duration=14.741502221s
Nov 15 10:25:35 prometheus: level=info ts=2019-11-15T18:25:35.795Z caller=head.go:598 component=tsdb msg="head GC completed" duration=22.593533ms
Nov 15 10:25:36 prometheus: level=info ts=2019-11-15T18:25:36.796Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50E9ZP9F1ZSY2YD30GEHK duration=732.572029ms
Nov 15 10:25:37 prometheus: level=error ts=2019-11-15T18:25:37.062Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50E9ZP9F1ZSY2YD30GEHK:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:38 prometheus: level=info ts=2019-11-15T18:25:38.824Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50G8EYTYBPWWHWZ1RJV5A duration=761.592662ms
Nov 15 10:25:39 prometheus: level=error ts=2019-11-15T18:25:39.087Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50G8EYTYBPWWHWZ1RJV5A:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:41 prometheus: level=info ts=2019-11-15T18:25:41.816Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50K6ZJFJQ4217JW2E5XBV duration=729.049353ms
Nov 15 10:25:42 prometheus: level=error ts=2019-11-15T18:25:42.087Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50K6ZJFJQ4217JW2E5XBV:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:46 prometheus: level=info ts=2019-11-15T18:25:46.814Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50R37245ZE56EZNH0335G duration=727.559705ms
Nov 15 10:25:47 prometheus: level=error ts=2019-11-15T18:25:47.084Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50R37245ZE56EZNH0335G:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:55 prometheus: level=info ts=2019-11-15T18:25:55.807Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR510WDE18F41GJBRVXBEDE duration=722.624112ms
Nov 15 10:25:55 prometheus: level=error ts=2019-11-15T18:25:56.080Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR510WDE18F41GJBRVXBEDE:mmap files: mmap: cannot allocate memory]"

prometheus service:


[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

I have installed the influxdb grafana on the same server.

Please let me know what exactly the issue is?

Thanks in advance

Upvotes: 2

Views: 3922

Answers (1)

mhansen
mhansen

Reputation: 1141

Are you running 32-bit prometheus? You're probably running out of address space. I don't think Prometheus plans to make this work on 32-bit systems, it simply runs out of 4GB address space with all the mmaping happening.

More details at https://github.com/prometheus/prometheus/issues/4392#issuecomment-433721793

Upvotes: 3

Related Questions