CODEWITHSUNDEEP
gitlabprometheusprometheus-alertmanagerprometheus-operatorkube-prometheus-stack
Abdullah Khawer
Abdullah Khawer

Reputation: 5678

Prometheus error: scraping GitLab targets: non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target

I'm using kube-prometheus-stack via Helm release on my AWS EKS cluster and I have started seeing the following error via alert manager:

enter image description here

In Prometheus logs, I see the following error: fallback_media_type="" err="non-compliant scrape target sending blank Content-Type and no fallback_scrape_protocol specified for target"

Helm release version: 67.4.0

Prometheus Image Version: v3.0.1

I have checked both the components on GitLab and they are working fine. It has something to do with Prometheus scraping.

How can I fix this?

Upvotes: 0

Views: 126

Answers (1)

Abdullah Khawer
Abdullah Khawer

Reputation: 5678

I have found the solution. I upgraded the kube-prometheus-stack Helm release to 68.1.0 and Prometheus image v3.1.0 and after that, I updated the Prometheus configuration and added fallback_scrape_protocol: PrometheusText0.0.4 in the following 2 jobs as follows:

...
- job_name: gitlab_exporter_database
  fallback_scrape_protocol: PrometheusText0.0.4
  metrics_path: /database
  static_configs:
  - targets:
    - __REDACTED__
- job_name: gitlab_exporter_sidekiq
  fallback_scrape_protocol: PrometheusText0.0.4
  metrics_path: /sidekiq
  static_configs:
  - targets:
    - __REDACTED__
- job_name: gitaly
...

After that, it worked.

Reference for solution is here.

Upvotes: 1

Related Questions