CODEWITHSUNDEEP
javagoprometheusmesosmarathon
stuart
stuart

Reputation: 1070

Prometheus dynamic port service discovery with Marathon/Mesos

I've been unable to find an answer to this after days of Googling. I have a service running in Marathon/Mesos. I have a Prometheus cluster scraping metrics. My Marathon metrics port config looks like this:

{
  "containerPort": 8081,
  "hostPort": 0,
  "servicePort": 31301,
  "protocol": "tcp",
  "labels": {
    "metrics": "/metrics"
  }
}

Prometheus, configured with just a boilerplate marathon-sd config, successfully finds this target, but it then listens for metrics on: __address__ = [NodeIP]:31301; so it's listening on the host's IP with the service port, rather than the dynamically assigned host port, while the service port only matters for Marathon-LB.

I know that Marathon defines the environment variable $PORT0 in the container to be the host port, however I can't figure out how to access this from the Prometheus SD config, nor how to access other fields dynamically configured by Marathon like Endpoints. Does anyone have any suggestions? I can't/don't want to assign a static hostPort because I have more containers than there are physical nodes in the cluster, as is sort of the point with container orchestration.

Upvotes: 2

Views: 650

Answers (1)

Andrey Dyatlov
Andrey Dyatlov

Reputation: 1668

It is a known bug in Prometheus: it uses the servicePort Marathon app definition property instead of the hostPort one. It is fixed in the v2.6.0.

Upvotes: 1

Related Questions