CODEWITHSUNDEEP
spring-bootprometheus
Technext
Technext

Reputation: 8107

Configuring Prometheus for Spring Boot health check monitoring

We have few Spring Boot applications that have health check implemented. The response for those checks was changed to JSON format based on @Thiru's suggestion. I now get the following response:

enter image description here

Prometheus server is running on Ubuntu instance. Spring boot services that have to be monitored are running on Windows server 2016. I have installed blackbox-exporter (version 0.12.0.windows-amd64) on Windows server after seeing this post.

Following change was done on client (Windows server with IP: 172.16.x.yz) side in blackbox.yml:

modules:
  http_2xx:
    prober: http
    http:
  http_post_2xx:
    prober: http
    timeout: 5s
    http:
      method: POST
      headers:
        Content-Type: application/json
      body: '{"status": "UP"}'
  ...
  ...

On Prometheus server, following is the content of prometheus.yml:

...
...
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      #module: [http_2xx]  # Look for a HTTP 200 response.
      module: [http_post_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://172.16.x.yz:6300/serviceA/actuator/health
        - http://172.16.x.yz:6340/serviceB/actuator/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 172.16.x.yz:9115  # The blackbox exporter's real hostname:port.

After making above changes on client and server, when i restart both the blackbox-exporter and prometheus, i see that Prometheus always shows State as UP even if the two services that the exporter is monitoring go down. Seems Prometheus is displaying the status of the blackbox-exporter and not for the services. Any suggestion how i can fix that?

enter image description here

Upvotes: 4

Views: 18500

Answers (2)

Waiyan Wh
Waiyan Wh

Reputation: 1

As my opinion, prometheus targets actually show that the scrape target is in up or down state. It's not about the healthcheck endpoint state. You can only check the healthcheck endpoint status in probe_success metrics( 1 is up and 0 is down ) . You can check the following screenshots. Sorry for my bad english.

Targets status

Real Status of endpoint

Upvotes: 0

Thiru
Thiru

Reputation: 2699

I would recommend to use some kind of tools which helps in exporting the spring health as prometheus exporter. Basically, exporter which helps in converting json data from an http url into prometheus metrics using jsonpath like:

Upvotes: 3

Related Questions