How do I access this Kubernetes service via kubectl proxy?

Question

I want to access my Grafana Kubernetes service via the kubectl proxy server, but for some reason it won't work even though I can make it work for other services. Given the below service definition, why is it not available on http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana?

grafana-service.yaml

apiVersion: v1
kind: Service
metadata:
  namespace: monitoring
  name: grafana
  labels:
    app: grafana
spec:
  type: NodePort
  ports:
  - name: web
    port: 3000
    protocol: TCP
    nodePort: 30902
  selector:
    app: grafana

grafana-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: monitoring
  name: grafana
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:4.1.1
        env:
        - name: GF_AUTH_BASIC_ENABLED
          value: "true"
        - name: GF_AUTH_ANONYMOUS_ENABLED
          value: "true"
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: user
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: password
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/grafana-storage
        ports:
        - name: web
          containerPort: 3000
        resources:
          requests:
            memory: 100Mi
            cpu: 100m
          limits:
            memory: 200Mi
            cpu: 200m
      - name: grafana-watcher
        image: quay.io/coreos/grafana-watcher:v0.0.5
        args:
          - '--watch-dir=/var/grafana-dashboards'
          - '--grafana-url=http://localhost:3000'
        env:
        - name: GRAFANA_USER
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: user
        - name: GRAFANA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: password
        resources:
          requests:
            memory: "16Mi"
            cpu: "50m"
          limits:
            memory: "32Mi"
            cpu: "100m"
        volumeMounts:
        - name: grafana-dashboards
          mountPath: /var/grafana-dashboards
      volumes:
      - name: grafana-storage
        emptyDir: {}
      - name: grafana-dashboards
        configMap:
          name: grafana-dashboards

The error I'm seeing when accessing the above URL is "no endpoints available for service "grafana"", error code 503.

Answer 1

With Kubernetes 1.10 the proxy URL should be slighly different, like this:

http://localhost:8080/api/v1/namespaces/default/services/SERVICE-NAME:PORT-NAME/proxy/ 

Ref: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#manually-constructing-apiserver-proxy-urls

Answer 2

The issue is that Grafana's port is named web, and as a result one needs to append :web to the kubectl proxy URL: http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana:web.

An alternative, is to instead not name the Grafana port, because then you don't have to append :web to the kubectl proxy URL for the service: http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana:web. I went with this option in the end since it's easier.

Answer 3

As Michael says, quite possibly your labels or namespaces are mismatching. However in addition to that, keep in mind that even when you fix the endpoint, the url you're after (http://localhost:8001/api/v1/proxy/namespaces/monitoring/services/grafana) might not work correctly.

Depending on your root_url and/or static_root_path grafana configuration settings, when trying to login you might get grafana trying to POST to http://localhost:8001/login and get a 404.

Try using kubectl port-forward instead:

kubectl -n monitoring port-forward [grafana-pod-name] 3000

then access grafana via http://localhost:3000/

https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/

Answer 4

There are a few factors that might be causing this issue.

1. The service expects to find one or more supporting endpoints, which it discovers through matching rules on the labels. If the labels don't align, then the service won't find endpoints, and the network gateway function performed by the service will result in 503.

2. The port declared by the POD and the process within the container are misaligned from the --target-port expected by the service.

Either one of these might generate the error. Let's take a closer look.

First, kubectl describe the service:

$ kubectl describe svc grafana01-grafana-3000
Name:           grafana01-grafana-3000
Namespace:      default
Labels:         app=grafana01-grafana
            chart=grafana-0.3.7
            component=grafana
            heritage=Tiller
            release=grafana01
Annotations:        <none>
Selector:       app=grafana01-grafana,component=grafana,release=grafana01
Type:           NodePort
IP:         10.0.0.197
Port:           <unset> 3000/TCP
NodePort:       <unset> 30905/TCP
Endpoints:      10.1.45.69:3000
Session Affinity:   None
Events:         <none>

Notice that my grafana service has 1 endpoint listed (there could be multiple). The error above in your example indicates that you won't have endpoints listed here.

Endpoints:      10.1.45.69:3000

Let's take a look next at the selectors. In the example above, you can see I have 3 selector labels on my service:

Selector:       app=grafana01-grafana,component=grafana,release=grafana01

I'll kubectl describe my pods next:

$ kubectl describe pod grafana
Name:       grafana01-grafana-1843344063-vp30d
Namespace:  default
Node:       10.10.25.220/10.10.25.220
Start Time: Fri, 14 Jul 2017 03:25:11 +0000
Labels:     app=grafana01-grafana
        component=grafana
        pod-template-hash=1843344063
        release=grafana01
...

Notice that the labels on the pod align correctly, hence my service finds pods which provide endpoints which are load balanced against by the service. Verify that this part of the chain isn't broken in your environment.

If you do find that the labels are correct, you may still have a disconnect in that the grafana process running within the container within the pod is running on a different port than you expect.

$ kubectl describe pod grafana
Name:       grafana01-grafana-1843344063-vp30d
...
Containers:
  grafana:
    Container ID:   docker://69f11b7828c01c5c3b395c008d88e8640c5606f4d865107bf4b433628cc36c76
    Image:      grafana/grafana:latest
    Image ID:       docker-pullable://grafana/grafana@sha256:11690015c430f2b08955e28c0e8ce7ce1c5883edfc521b68f3fb288e85578d26
    Port:       3000/TCP
    State:      Running
      Started:      Fri, 14 Jul 2017 03:25:26 +0000

If for some reason, your port under the container listed a different value, then the service is effectively load balancing against an invalid endpoint.

For example, if it listed port 80: Port: 80/TCP Or was an empty value Port:

Then even if your label selectors were correct, the service would never find a valid response from the pod and would remove the endpoint from the rotation.

I suspect your issue is the first problem above (mismatched label selectors).

If both the label selectors and ports align, then you might have a problem with the MTU setting between nodes. In some cases, if the MTU used by your networking layer (like calico) is larger than the MTU of the supporting network, then you'll never get a valid response from the endpoint. Typically, this last potential issue will manifest itself as a timeout rather than a 503 though.

Answer 5

Your Deployment may not have a label app: grafana, or be in another namespace. Could you also post the Deployment definition?