CODEWITHSUNDEEP
dockerapache-sparkkubernetes
Vinh Tran
Vinh Tran

Reputation: 189

Launching Spark 3.0 on Kubernetes -- Error Pulling image

I'm trying to launch spark on Kubernetes per the instructions on the official website: Spark Kubernetes. I followed the instructions to create an image using their provided script as followings:

./bin/docker-image-tool.sh -r <repo> -t my-tag -p ./kubernetes/dockerfiles/spark/bindings/python/Dockerfile build

For the <repo> I used pyspark. My spark submit for the Kubernetes is as follows:

./bin/spark-submit \
    --master k8s://https://<ipaddress>:<port>\
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=pyspark/spark-py:my-tag \
    --conf spark.kubernetes.file.upload.path=/usr/local/spark/examples \
    local:///usr/local/spark/examples/spark-examples_2.12-3.0.0.jar

The kubernetes set ups looks ok, however the pods are having issues with the images it pulled which results in restart. I'm not sure if I have the right nomenclature of the image.

Upvotes: 0

Views: 1359

Answers (1)

Tarun Khosla
Tarun Khosla

Reputation: 1362

The example uses pyspark/spark-py:my-tag as container image which should be present , As this is not present you are getting the error as Image Pull Error. You can push your image to dockerhub or your private registry and use it as example.com/repo/spark:v1.0.0 where example.com is your private registry and repo is the repository you have there.

Also when you run ./bin/docker-image-tool.sh -r <repo> -t my-tag build , you need to specify a repo. Your command should be like ./bin/docker-image-tool.sh -r docker.io/myrepo -t v2.3.0 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build

If you are using minikube then building images will do so directly into minikube's Docker daemon.There is no need to push the images into minikube in that case, they'll be automatically available when running applications inside the minikube cluster.

Upvotes: 2

Related Questions