Dung-Tri LE
Reputation: 31
Traefik on a rpi kubernetes node return 404 page not found
I try to made a first experience on kubernetes by practice.
kubernetes v1.9 has been setup on 5 raspberry pi mounted as cluster.
OS : hypriot v1.4
host / static ip configured / raspberry hardware version :
- master: 192.168.1.230 / rpi v3
- node01: 192.168.1.231 / rpi v3
- node02: 192.168.1.232 / rpi v3
- node03: 192.168.1.233 / rpi v2
- node04: 192.168.1.234 / rpi v2
For the pod network I choose Weave Net. Traefik has been installed in the node01 as load balancer to access my service from outside. I ssh the master and use these commands to install it (origin: https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/) :
$ kubectl apply -f https://raw.githubusercontent.com/hypriot/rpi-traefik/master/traefik-k8s-example.yaml
$ kubectl label node node01 nginx-controller=traefik
All system pods are running.
$ kubectl get pods --all-namespaces
kube-system etcd-master 1/1 Running 5 22h
kube-system kube-apiserver-master 1/1 Running 40 13h
kube-system kube-controller-manager-master 1/1 Running 10 13h
kube-system kube-dns-7b6ff86f69-x58pj 3/3 Running 9 23h
kube-system kube-proxy-5bqwh 1/1 Running 2 15h
kube-system kube-proxy-kngp9 1/1 Running 2 16h
kube-system kube-proxy-n85xl 1/1 Running 5 23h
kube-system kube-proxy-ncg2k 1/1 Running 2 15h
kube-system kube-proxy-qbfcf 1/1 Running 2 21h
kube-system kube-scheduler-master 1/1 Running 5 22h
kube-system traefik-ingress-controller-9dc7454cc-7rhpf 1/1 Running 1 14h
kube-system weave-net-6mvc6 2/2 Running 31 15h
kube-system weave-net-8hff9 2/2 Running 31 15h
kube-system weave-net-9kwgr 2/2 Running 31 21h
kube-system weave-net-llgrk 2/2 Running 41 22h
kube-system weave-net-s2h62 2/2 Running 29 16h
The issue is when I try to connect to the node01 by using this url http://192.168.1.231/. I got a 404 page not found...
So I checked the log and figure out that they are a problem with the default account :
$ kubectl logs traefik-ingress-controller-9dc7454cc-7rhpf
ERROR: logging before flag.Parse: E1226 07:29:15.195193 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:kube-system:default" cannot list endpoints at the cluster scope
ERROR: logging before flag.Parse: E1226 07:29:15.422807 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:kube-system:default" cannot list secrets at the cluster scope
ERROR: logging before flag.Parse: E1226 07:29:15.915317 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list services at the cluster scope
ERROR: logging before flag.Parse: E1226 07:29:16.108385 1 reflector.go:199] github.com/containous/traefik/vendor/k8s.io/client-go/tools/cache/reflector.go:94: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:kube-system:default" cannot list ingresses.extensions at the cluster scope
Is it really a problem with the account system:serviceaccount:kube-system:default used? What account should I use instead of?
Thanks for helping.
Additional informations:
$ docker -v Docker version 17.03.0-ce, build 60ccb22
$ kubectl describe pods traefik-ingress-controller -n kube-system
Name: traefik-ingress-controller-9dc7454cc-7rhpf
Namespace: kube-system
Node: node01/192.168.1.231
Start Time: Mon, 25 Dec 2017 20:54:45 +0000
Labels: k8s-app=traefik-ingress-controller
pod-template-hash=587301077
Annotations: scheduler.alpha.kubernetes.io/tolerations=[
{
"key": "dedicated",
"operator": "Equal",
"value": "master",
"effect": "NoSchedule"
}
]
Status: Running
IP: 192.168.1.231
Controlled By: ReplicaSet/traefik-ingress-controller-9dc7454cc
Containers:
traefik-ingress-controller:
Container ID: docker://9e28800da6937a48aa20b5ef6526846b321a516ad20ee24ea3d32876f6769531
Image: hypriot/rpi-traefik
Image ID: docker-pullable://hypriot/rpi-traefik@sha256:ecdfcd94571ec8c121c20a6ec616d68aeaad93150a0717260196f813e31737d9
Ports: 80/TCP, 8888/TCP
Args:
--web
--web.address=localhost:8888
--kubernetes
State: Running
Started: Mon, 25 Dec 2017 22:24:33 +0000
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Mon, 25 Dec 2017 20:54:50 +0000
Finished: Mon, 25 Dec 2017 22:17:09 +0000
Ready: True
Restart Count: 1
Limits:
cpu: 200m
memory: 30Mi
Requests:
cpu: 100m
memory: 20Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4wzhl (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-4wzhl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4wzhl
Optional: false
QoS Class: Burstable
Node-Selectors: nginx-controller=traefik
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Name: traefik-ingress-controller-9dc7454cc-jszgz
Namespace: kube-system
Node: node01/
Start Time: Mon, 25 Dec 2017 18:28:21 +0000
Labels: k8s-app=traefik-ingress-controller
pod-template-hash=587301077
Annotations: scheduler.alpha.kubernetes.io/tolerations=[
{
"key": "dedicated",
"operator": "Equal",
"value": "master",
"effect": "NoSchedule"
}
]
Status: Failed
Reason: MatchNodeSelector
Message: Pod Predicate MatchNodeSelector failed
IP:
Controlled By: ReplicaSet/traefik-ingress-controller-9dc7454cc
Containers:
traefik-ingress-controller:
Image: hypriot/rpi-traefik
Ports: 80/TCP, 8888/TCP
Args:
--web
--web.address=localhost:8888
--kubernetes
Limits:
cpu: 200m
memory: 30Mi
Requests:
cpu: 100m
memory: 20Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4wzhl (ro)
Volumes:
default-token-4wzhl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4wzhl
Optional: false
QoS Class: Burstable
Node-Selectors: nginx-controller=traefik
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
$ kubectl describe pods weave-net-9kwgr -n kube-system
Name: weave-net-llgrk
Namespace: kube-system
Node: master/192.168.1.230
Start Time: Mon, 25 Dec 2017 13:33:40 +0000
Labels: controller-revision-hash=2209123374
name=weave-net
pod-template-generation=1
Annotations: <none>
Status: Running
IP: 192.168.1.230
Controlled By: DaemonSet/weave-net
Containers:
weave:
Container ID: docker://7824b8b02f1a8f5a53d7f27f0c12b44f73a4b666a694b974142f974294bedd6c
Image: weaveworks/weave-kube:2.1.3
Image ID: docker-pullable://weaveworks/weave-kube@sha256:07a3d56b8592ea3e00ace6f2c3eb7e65f3cc4945188a9e2a884b8172e6a0007e
Port: <none>
Command:
/home/weave/launch.sh
State: Running
Started: Tue, 26 Dec 2017 00:13:58 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 26 Dec 2017 00:08:38 +0000
Finished: Tue, 26 Dec 2017 00:08:50 +0000
Ready: True
Restart Count: 37
Requests:
cpu: 10m
Liveness: http-get http://127.0.0.1:6784/status delay=30s timeout=1s period=10s #success=1 #failure=3
Environment:
HOSTNAME: (v1:spec.nodeName)
Mounts:
/host/etc from cni-conf (rw)
/host/home from cni-bin2 (rw)
/host/opt from cni-bin (rw)
/host/var/lib/dbus from dbus (rw)
/lib/modules from lib-modules (rw)
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-mx5jk (ro)
/weavedb from weavedb (rw)
weave-npc:
Container ID: docker://b199904c10ed34501748c25e13862113aeb32c7779b0797d72c95f9e9d868331
Image: weaveworks/weave-npc:2.1.3
Image ID: docker-pullable://weaveworks/weave-npc@sha256:f35eb8166d7dae3fa7bb4d9892ab6dc8ea5c969f73791be590a0a213767c0f07
Port: <none>
State: Running
Started: Mon, 25 Dec 2017 22:24:32 +0000
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Mon, 25 Dec 2017 20:54:30 +0000
Finished: Mon, 25 Dec 2017 22:17:09 +0000
Ready: True
Restart Count: 4
Requests:
cpu: 10m
Environment:
HOSTNAME: (v1:spec.nodeName)
Mounts:
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-mx5jk (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
weavedb:
Type: HostPath (bare host directory volume)
Path: /var/lib/weave
HostPathType:
cni-bin:
Type: HostPath (bare host directory volume)
Path: /opt
HostPathType:
cni-bin2:
Type: HostPath (bare host directory volume)
Path: /home
HostPathType:
cni-conf:
Type: HostPath (bare host directory volume)
Path: /etc
HostPathType:
dbus:
Type: HostPath (bare host directory volume)
Path: /var/lib/dbus
HostPathType:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType:
weave-net-token-mx5jk:
Type: Secret (a volume populated by a Secret)
SecretName: weave-net-token-mx5jk
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: :NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events: <none>
root@master:/home/pirate# kubectl describe pods weave-net-9kwgr -n kube-system
Name: weave-net-9kwgr
Namespace: kube-system
Node: node01/192.168.1.231
Start Time: Mon, 25 Dec 2017 14:50:37 +0000
Labels: controller-revision-hash=2209123374
name=weave-net
pod-template-generation=1
Annotations: <none>
Status: Running
IP: 192.168.1.231
Controlled By: DaemonSet/weave-net
Containers:
weave:
Container ID: docker://92e31f645b4dcd41e4d8189a6f67fa70a395971e071d635dc4c4208b8d1daf63
Image: weaveworks/weave-kube:2.1.3
Image ID: docker-pullable://weaveworks/weave-kube@sha256:07a3d56b8592ea3e00ace6f2c3eb7e65f3cc4945188a9e2a884b8172e6a0007e
Port: <none>
Command:
/home/weave/launch.sh
State: Running
Started: Tue, 26 Dec 2017 00:13:39 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 26 Dec 2017 00:08:17 +0000
Finished: Tue, 26 Dec 2017 00:08:28 +0000
Ready: True
Restart Count: 29
Requests:
cpu: 10m
Liveness: http-get http://127.0.0.1:6784/status delay=30s timeout=1s period=10s #success=1 #failure=3
Environment:
HOSTNAME: (v1:spec.nodeName)
Mounts:
/host/etc from cni-conf (rw)
/host/home from cni-bin2 (rw)
/host/opt from cni-bin (rw)
/host/var/lib/dbus from dbus (rw)
/lib/modules from lib-modules (rw)
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-mx5jk (ro)
/weavedb from weavedb (rw)
weave-npc:
Container ID: docker://ddd86bef74d3fd40134c8609551cc07658aa62a2ede7ce51aec394001049e96d
Image: weaveworks/weave-npc:2.1.3
Image ID: docker-pullable://weaveworks/weave-npc@sha256:f35eb8166d7dae3fa7bb4d9892ab6dc8ea5c969f73791be590a0a213767c0f07
Port: <none>
State: Running
Started: Mon, 25 Dec 2017 22:24:32 +0000
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Mon, 25 Dec 2017 20:54:30 +0000
Finished: Mon, 25 Dec 2017 22:17:09 +0000
Ready: True
Restart Count: 2
Requests:
cpu: 10m
Environment:
HOSTNAME: (v1:spec.nodeName)
Mounts:
/run/xtables.lock from xtables-lock (rw)
/var/run/secrets/kubernetes.io/serviceaccount from weave-net-token-mx5jk (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
weavedb:
Type: HostPath (bare host directory volume)
Path: /var/lib/weave
HostPathType:
cni-bin:
Type: HostPath (bare host directory volume)
Path: /opt
HostPathType:
cni-bin2:
Type: HostPath (bare host directory volume)
Path: /home
HostPathType:
cni-conf:
Type: HostPath (bare host directory volume)
Path: /etc
HostPathType:
dbus:
Type: HostPath (bare host directory volume)
Path: /var/lib/dbus
HostPathType:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType:
weave-net-token-mx5jk:
Type: Secret (a volume populated by a Secret)
SecretName: weave-net-token-mx5jk
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: :NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events: <none>
Upvotes: 0
Views: 2031
Answers (1)
Timo Reimann
Reputation: 9837
Your Traefik service account is missing proper RBAC privileges. By default, no application may access any Kubernetes API.
You have to make sure that the necessary rights are granted. Please check our Kubernetes guide for details.
Upvotes: 2
Related Questions
- Traefik version 2 only shows 404 or no website at all
- Traefik configuration with Docker not working
- Traefik v2 with Docker shows 404 page not found
- Traefik v2 http 404 redirect to external
- why the traefik did not match the url by the path
- Traefik shows 404 to everywhere except dashboard - Metallb
- Kubernetes:Traefik (v2.2) dashboard returning "404 page not found"
- Why does Traefik v2 response 404 only over http
- One Traefik Pod in Kubernetes fails with error: 'command traefik error: field not found, node: redirect'
- Why is traefik not detecting my containers