CODEWITHSUNDEEP
apache-sparkgoogle-cloud-platformgoogle-kubernetes-enginegoogle-cloud-dataproc
gagan
gagan

Reputation: 355

Dataproc on GKE doesn't use image when submitting the job

Why does Dataproc on GKE not support Docker images for jobs submission but uses raw code for python/R and jar for others. Shouldn't providing container image be the ideal way to submit job to GKE be it ordinary job or Dataproc job? Is this in plan or a work in progress?

Ref: https://cloud.google.com/dataproc/docs/concepts/jobs/dataproc-gke

Upvotes: 2

Views: 403

Answers (1)

Igor Dvorzhak
Igor Dvorzhak

Reputation: 4455

As mentioned in the question comments, the main intent behind current iteration of Dataproc on GKE is to provide the same user experience as Dataproc on GCE. That's why it supports the same API which doesn't use Docker images to execute Spark jobs and there are no near-term plans to add support for this. That said, in the future Dataproc may decide to support some kind of containers as a first class way of submitting jobs (in addition to jars, etc).

Upvotes: 1

Related Questions