CODEWITHSUNDEEP
spring-bootprometheus
b15
b15

Reputation: 2361

"INVALID" is not a valid start token

Can't for the life of me figure out why I'm getting this error for one of the applications I'm trying to scrape. I have the following prometheus.yml:

# prometheus.yml
global:
    scrape_interval: 5s
    external_labels:
        monitor: 'my-monitor'

scrape_configs:
    - job_name: 'prometheus'
      metrics_path: /metrics
      static_configs:
          - targets: ['localhost:9090']
    - job_name: 'app'
      metrics_path: /custom_metrics
      static_configs:
          - targets: ['host.docker.internal:8888']
      basic_auth:
          username: user
          password: pass
    - job_name: 'otherapp'
      metrics_path: /custom_metrics
      static_configs:
          - targets: ['host.docker.internal:8081']
      basic_auth:
          username: admin
          password: admin

I'm running prometheus with docker and both applications are running on my local host. 'app' is being scraped fine but I'm seeing the following in the targets section of the Prometheus UI:

prometheus ui

I've run the above configuration against promtool check config with no complaints and I've also run the metrics from both of my applications through promtool check metrics with no complaints other than warnings about missing 'help'

I can hit the urls from the screen shot locally just fine and use the basic_auth username and pass from the config to see the metrics.

At this point I'm at a loss as to how to proceed. Is there something wrong with my config? Is there something that might be wrong with my metrics that promtool wouldn't catch?

Upvotes: 0

Views: 3710

Answers (2)

heat1q
heat1q

Reputation: 11

If anyone else encounters this issue, check if your app gzip compresses the metrics response. I didn't notice this, since curl does not add the Accept-Encoding header to the request by default.

After adding the encoding header, I realized that the metrics were actually compressed and the promtool check failed:

curl -s -H 'Accept-Encoding: gzip,deflate' http://localhost:8000/metrics | promtool check metrics

I removed the gzip compression from the /metrics endpoint and it worked.

Upvotes: 1

b15
b15

Reputation: 2361

One of my applications was configuring Spring Security to use basic auth and the other wasn't. The one that wasn't was showing

“INVALID” is not a valid start token

Until I added the following to my security configuration

.httpBasic()

I was surprised an authentication issue would have surfaced this way based on everything I had read about this error so maybe this will help others who are also surprised by this in the future.

Upvotes: 1

Related Questions