IBM Cloud Private 2.1.0 Console doesn't work after restart

Question

IBM Cloud Private 2.1 installation, configuration is a demo config, 1 x master and proxy and 1 x worker node.

Installation runs through fine and once completed I am able to logon to the GUI. After a reboot the GUI doesn't come up and I cant login in.

I have disabled the firewall just in case by still no luck.

--> ran the following command to get status of PODS (as WebUI is unavailable)

kubectl -s 127.0.0.1:8888 -n kube-system get pods

-- > output

[root@cpmaster ~]# kubectl -s 127.0.0.1:8888 -n kube-system get pods
NAME                                               READY     STATUS             RESTARTS   AGE
calico-node-ppc64le-496xm                          2/2       Running            10         15d
filebeat-ds-ppc64le-qssl2                          1/1       Running            5          15d
k8s-etcd-X.X.X.X                                   1/1       Running            5          15d
k8s-mariadb-X.X.X.X                                1/1       Running            5          15d
k8s-master-X.X.X.X                                 2/3       CrashLoopBackOff   559        4m
k8s-proxy-X.X.X.X                                  1/1       Running            5          15d
metering-reader-ppc64le-gcc8w                      1/1       Running            5          15d
monitoring-prometheus-nodeexporter-ppc64le-zq8ls   1/1       Running            5          15d

Looks like I have a problem with the k2s-master POD

Answer 1

I do not claim to be knowledgeable in this area but I got this solution from the development - which people tested and reported that it works for them.

The issue as per the developer is:

1. icp-ds sometimes does not startup correctly after a reboot, which is due to an ipaddress allocation issue by Calico in time.

2. icp-ds readiness probe times out too soon with a result that it forces out the pod to go into loop, repeatedly restarting. To workaround this - increase the probe timeout and period in the icp-ds statefulset.

Now I got the following 2 commands from him to do this in an automated fashion.

Go to the master node. 1. Run kubectl -s http://127.0.0.1:8888 get pods --all-namespaces

You should see kube-system icp-ds-0 0/1 Running 0 34m running. This is good things.

Then we will run a command to patch to add / modify the following values.

initialDelaySeconds: 180 
periodSeconds: 60 
failureThreshold: 20

The 2nd command:

# kubectl -s http://127.0.0.1:8888 patch StatefulSet icp-ds -p '{"spec":{"template":{"spec":{"containers":[{"name":"icp-ds", "readinessProbe":{"periodSeconds":60, "initialDelaySeconds":180, "failureThreshold":20, "timeoutSeconds":5}}]}}}}' -n kube-system

Alternatively, you could also use kubectl -s http://127.0.0.1:8888 edit sts/icp-ds and it will open a vi editor and you have to manually go and modify the values.

Thanks Chaitnya K for the workaround - which I can use in the script and hopefully - this will get fixed in upcoming releases.

This issue was in 2.1.0 and still exists in 2.1.0.1.

Tip: (I did not know this) - port 8001 is ssl port and port 8888 is http port. So, when you can't have access to Web UI, use port 8888 as explained above and fix this issue and then have access to the UI.

Answer 2

Please review the known issue and limitation page and try the workaround, it may match your issue.

In the known issues link below --check for the "502 Bad Gateway Error" by pinging icp-ds-0.

https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0/getting_started/known_issues.html

-> Kubernetes controller manager fails to start after a master or cluster restart

Thank you!

Answer 3

You can see where ui pod is running or not:

kubectl -s 127.0.0.1:8888 -n kube-system get pods |grep platform-ui

Sometimes, it takes a while for the the pod to come back after machine reboot.

You can also check the logs for the platform-ui container to trace the issue:

#enter code here # get container-id
docker ps | grep platform-ui
# see the logs
docker logs container-id