How to get access to the Heapster's Grafana dashboard after clean Kubernetes 1.5.0 installation using Rancher 1.5.1

Question

I'm deploying my test Kubernetes cluster on AWS using Rancher 1.5.1.

I first deploy Rancher and ensure that there are no errors after hosts registration. I then install Kubernetes using "Kubernetes" entry in the Rancher's catalog. That succeeds seemingly without any errors and I get access to my 1.5.0 Kubernetes:

kubectl version
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.4", GitCommit:"7243c69eb523aa4377bce883e7c0dd76b84709a1", GitTreeState:"clean", BuildDate:"2017-03-08T02:50:34Z", GoVersion:"go1.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-115+611cbb22703182", GitCommit:"611cbb22703182611863beda17bf9f3e90afa148", GitTreeState:"clean", BuildDate:"2017-01-13T18:03:00Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

As I understand, the Heapster with InfluxDB and Grafana dashboard is part of the default Kubernetes installation now. All Heapster, InfluxDB and Grafana related pods show no errors nether in logs nor in status and seem to run successfully:

kubectl -n kube-system describe rs heapster-3467702493
Name:       heapster-3467702493
Namespace:  kube-system
Image(s):   gcr.io/google_containers/heapster:v1.2.0
Selector:   k8s-app=heapster,pod-template-hash=3467702493,version=v6
Labels:     k8s-app=heapster
        pod-template-hash=3467702493
        version=v6
Replicas:   1 current / 1 desired
Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
No events.

kubectl -n kube-system get pods
NAME                                    READY     STATUS             RESTARTS   AGE
default-http-backend-w9td0              1/1       Running            0          1d
heapster-3467702493-b28jm               1/1       Running            0          1d
influxdb-grafana-876329878-5qpsd        2/2       Running            0          1d
kube-dns-1208858260-zb7d9               4/4       Running            0          1d
kubernetes-dashboard-2492700511-8g3bj   1/1       Running            0          1d


kubectl -n kube-system describe pod influxdb-grafana-876329878-5qpsd
Name:       influxdb-grafana-876329878-5qpsd
Namespace:  kube-system
Node:       euir1a-dclus11.qiotec-internal.com/10.11.4.172
Start Time: Tue, 14 Mar 2017 14:48:05 +0100
Labels:     name=influx-grafana
        pod-template-hash=876329878
Status:     Running
IP:     10.42.35.83
Controllers:    ReplicaSet/influxdb-grafana-876329878
Containers:
  influxdb:
    Container ID:   docker://49ad7e2033d9116cc98d1e7c8cd6e20c305179d68804b762bb19592fefa59b3e
    Image:      docker.io/kubernetes/heapster_influxdb:v0.5
    Image ID:       docker-pullable://kubernetes/heapster_influxdb@sha256:24de37030e0da01c39b8863231b70f359e1fe6d4449505da03e2e7543bb068cb
    Port:
    State:      Running
      Started:      Tue, 14 Mar 2017 14:48:29 +0100
    Ready:      True
    Restart Count:  0
    Volume Mounts:
      /data from influxdb-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from io-rancher-system-token-zgrrs (ro)
    Environment Variables:  <none>
  grafana:
    Container ID:   docker://bdb4e381f0cd05df0a2d1c7dffb52b3e6e724a27999e039c5399fef391fd6d32
    Image:      gcr.io/google_containers/heapster_grafana:v2.6.0-2
    Image ID:       docker-pullable://gcr.io/google_containers/heapster_grafana@sha256:208c98b77d4e18ad7759c0958bf87d467a3243bf75b76f1240a577002e9de277
    Port:
    State:      Running
      Started:      Tue, 14 Mar 2017 14:48:41 +0100
    Ready:      True
    Restart Count:  0
    Volume Mounts:
      /var from grafana-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from io-rancher-system-token-zgrrs (ro)
    Environment Variables:
      INFLUXDB_SERVICE_URL:     http://monitoring-influxdb.kube-system.svc.cluster.local:8086
      GF_AUTH_BASIC_ENABLED:        false
      GF_AUTH_ANONYMOUS_ENABLED:    true
      GF_AUTH_ANONYMOUS_ORG_ROLE:   Admin
      GF_SERVER_ROOT_URL:       /
Conditions:
  Type      Status
  Initialized   True
  Ready     True
  PodScheduled  True
Volumes:
  influxdb-storage:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  grafana-storage:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  io-rancher-system-token-zgrrs:
    Type:   Secret (a volume populated by a Secret)
    SecretName: io-rancher-system-token-zgrrs
QoS Class:  BestEffort
Tolerations:    <none>
No events.

The strange thing I can only see here, is that "Port:" entries are empty (instead of having some value as I'd expected).

Here is part of the influxdb-grafana-876329878-5qpsd log. It seems that it starts successfully on port 3000:

....
2017/03/14 13:48:42 [I] Migrator: exec migration id: create index UQE_dashboard_snapshot_delete_key - v5
2017/03/14 13:48:42 [I] Migrator: exec migration id: create index IDX_dashboard_snapshot_user_id - v5
2017/03/14 13:48:42 [I] Migrator: exec migration id: alter dashboard_snapshot to mediumtext v2
2017/03/14 13:48:42 [I] Migrator: exec migration id: create quota table v1
2017/03/14 13:48:42 [I] Migrator: exec migration id: create index UQE_quota_org_id_user_id_target - v1
2017/03/14 13:48:42 [I] Created default admin user: admin
2017/03/14 13:48:42 [I] Listen: http://0.0.0.0:3000
.Grafana is up and running.
Creating default influxdb datasource...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   272  100    37  100   235   2972  18877 --:--:-- --:--:-- --:--:-- 19583
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Set-Cookie: grafana_sess=3063bf504a9ec00a; Path=/; HttpOnly
Date: Tue, 14 Mar 2017 13:48:43 GMT
Content-Length: 37

{"id":1,"message":"Datasource added"}
Importing default dashboards...
...

The services are also up:

kubectl -n kube-system get services
NAME                       CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
default-http-backend       10.43.159.75    <none>        80/TCP              1d
heapster                   10.43.77.135    <none>        80/TCP              1d
kube-dns                   10.43.0.10      <none>        53/UDP,53/TCP       1d
kubernetes-dashboard       10.43.202.63    <none>        9090/TCP            1d
monitoring-grafana         10.43.186.82    <none>        80/TCP              1d
monitoring-influxdb        10.43.61.17     <none>        8086/TCP,8083/TCP   1d

At the end, I cannot connect to the Grafana dashboard neither via Rancher load balancer mapping to the monitoring-grafana port 80 (I get 503 "Service not found" error) nor via Kubernetes port forwarding (connection times out)

My cluster-info output:

kubectl cluster-info
Kubernetes master is running at https://my-rancher-host:8080/r/projects/1a5/kubernetes
KubeDNS is running at https://my-rancher-host:8080/r/projects/1a5/kubernetes/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://my-rancher-host:8080/r/projects/1a5/kubernetes/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

is different from the one demonstrated in this Heapster video: https://www.youtube.com/watch?v=xSMNR2fcoLs

Can anybody give a hint about what can be wrong and how can I connect to the Heapster's Grafana dashboard?

Answer 1

Yes, almost always all HTTP services should have an ingress resource

Answer 2

It appears I had to add a correct ingress to the system:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: monitoring-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: monitoring.kube-system.mykubernetes.host
    http:
      paths:
      - path:
        backend:
          serviceName: monitoring-grafana
          servicePort: 80