Run Spark example on Kubernetes failed

Question

• My Mac OS/X Version : 10.15.3
• Minikube Version: 1.9.2

I start the minikube use the following command without any extra configuration.

minikube start --driver=virtualbox
--image-repository='registry.cn-hangzhou.aliyuncs.com/google_containers' --cpus 4 --memory 4096 --alsologtostderr

And I download spark-2.4.5-bin-hadoop2.7 from the Spark official website and build spark images by the following command

eval $(minikube docker-env)
./bin/docker-image-tool.sh -m -t 2.4.5 build

then I run Spark-pi using the follwing command within my local machine where store the Spark 2.4.5.

kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=admin --serviceaccount=default:spark --namespace=default
./bin/spark-submit \
--master k8s://https://192.168.99.104:8443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.executor.instances=1 \
--conf spark.kubernetes.container.image=spark:2.4.5 \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar

I get the following error

the full log can be found at full log

Can anyone explain this error and how to solve it?

Answer 1

Another spark image (from gcr.io/spark-operator/spark) worked for me, without downgrading the version of Kubernetes.

bin/spark-submit \
--master k8s://https://192.168.99.100:8443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.driver.cores=1 \
--conf spark.driver.memory=512m \
--conf spark.executor.instances=2 \
--conf spark.executor.memory=512m \
--conf spark.executor.cores=1 \
--conf spark.kubernetes.container.image=gcr.io/spark-operator/spark:v2.4.5 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar

Answer 2

Please check the Kubernetes version you launched with Minikube.

Spark v2.4.5 fabric8 Kubernetes client v4.6.1 is compatible with Kubernetes API up to 1.15.2 (refer answer).

You can launch the specific Kubernetes API version with Minikube by adding --kubernetes-version flag to minikube start command (refer docs).

Also the issue might be caused by OkHttp library bug described in the comment of this qustion.