CODEWITHSUNDEEP
apache-sparkkubernetes
Mike S
Mike S

Reputation: 1603

Some spark-submit config options not reflected in k8s pod

I'm using spark-submit to create a spark driver pod on my k8s cluster. When I run 

bin/spark-submit 
--master k8s://https://my-cluster-url:443
--deploy-mode cluster 
--name spark-test 
--class com.my.main.Class 
--conf spark.executor.instances=3 
--conf spark.kubernetes.allocation.batch.size=3 
--conf spark.kubernetes.namespace=my-namespace 
--conf spark.kubernetes.container.image.pullSecrets=my-cr-secret 
--conf spark.kubernetes.container.image.pullPolicy=Always 
--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.my-pvc.mount.path=/var/service/src/main/resources/ 
--conf spark.kubernetes.container.image=my-registry.io/spark-test:test-3.0.0 
local:///var/service/my-service-6.3.0-RELEASE.jar

spark-submit successfully creates a pod in my k8s cluster. However, many of the config options I specified are not seen. For example, the pod does not have a volume mounted at /var/service/src/main/resources/ despite the existence of a persistentVolumeClaim on the cluster called my-pvc. Further, the pod has not been given the specified image pull secret my-cr-secret, causing an ImagePullBackOff error. On the other hand, the pod is properly created in the my-namespace namespace and the pull policy Always.

I have attempted this using spark 3.0.0 and 2.4.5

Why are some config options not reflected in the pod created on my cluster?

Upvotes: 0

Views: 784

Answers (1)

Mike S
Mike S

Reputation: 1603

Figured out the issue:

I currently have spark 2.3.1 installed locally and the variable SPARK_HOME points to /usr/local/spark. For this current project I downloaded a distribution of spark 2.4.5. I was in the 2.4.5 directory and running bin/spark-submit, which should have (as far as I can tell) pointed to the spark-submit bundled in 2.4.5. However, running bin/spark-submit --version revealed that the version being run was 2.3.1. The configurations that were being ignored in my question above were not available in 2.3.1.

Simply changing SPARK_HOME to the new directory fixed the issue

Upvotes: 1

Related Questions