CODEWITHSUNDEEP
kuberneteskubeflow
João Areias
João Areias

Reputation: 1428

Kubeflow pipeline fail to create container

I'm running Kubeflow in a local machine that I deployed with multipass using these steps but when I tried running my pipeline, it got stuck with the message ContainerCreating. When I ran kubectl describe pod train-pipeline-msmwc-1648946763 -n kubeflow I found this on the Events part of the describe:

Events:
  Type     Reason       Age                    From                  Message
  ----     ------       ----                   ----                  -------
  Warning  FailedMount  7m12s (x51 over 120m)  kubelet, kubeflow-vm  Unable to mount volumes for pod "train-pipeline-msmwc-1648946763_kubeflow(45889c06-87cf-4467-8cfa-3673c7633518)": timeout expired waiting for volumes to attach or mount for pod "kubeflow"/"train-pipeline-msmwc-1648946763". list of unmounted volumes=[docker-sock]. list of unattached volumes=[podmetadata docker-sock mlpipeline-minio-artifact pipeline-runner-token-dkvps]
  Warning  FailedMount  2m22s (x67 over 122m)  kubelet, kubeflow-vm  MountVolume.SetUp failed for volume "docker-sock" : hostPath type check failed: /var/run/docker.sock is not a socket file

Looks to me like there is a problem with my deployment, but I'm new to Kubernetes and can't figure out what I supposed to do right now. Any idea on how to solve this? I don't know if it helps but I'm pulling the containers from a private docker registry and I've set up the secret according to this.

Upvotes: 3

Views: 1920

Answers (2)

Ali Akbari
Ali Akbari

Reputation: 51

You don't need to use docker. In fact the problem is with workflow-controller-configmap in kubeflow name space. You can edit it with

kubectl edit configmap workflow-controller-configmap -n kubeflow

and change containerRuntimeExecutor: docker to containerRuntimeExecutor: pns. Also you can change some of the steps and install kubeflow 1.3 in mutlitpass 1.21 rather than 1.15. Do not use kubelfow add-on (at least didn't work for me). You need kustomize 3.2 to create manifests as they mentioned in https://github.com/kubeflow/manifests#installation.

Upvotes: 5

João Areias
João Areias

Reputation: 1428

There was one step missing which is not mentioned in the tutorial, which is, I have to install docker. I've installed docker, rebooted the machine, and now everything works fine.

Upvotes: 3

Related Questions