CODEWITHSUNDEEP
gcloudgoogle-cloud-dataprocgoogle-apis-explorer
Georges Kohnen
Georges Kohnen

Reputation: 180

Unable to create Dataproc cluster using custom image

I am able to create a google dataproc cluster from the command line using a custom image:

gcloud beta dataproc clusters create cluster-name --image=custom-image-name

as specified in https://cloud.google.com/dataproc/docs/guides/dataproc-images, but I am unable to find information about how to do the same using the v1beta2 REST api in order to create a cluster from within airflow. Any help would be greatly appreciated.

Upvotes: 0

Views: 1044

Answers (1)

Dennis Huo
Dennis Huo

Reputation: 10687

Since custom images can theoretically reside in a different project if you grant read/use access of that custom image to whatever project service account you use for the Dataproc cluster, images currently always need a full URI, not just a short name.

When you use gcloud, there's syntactic sugar where gcloud will resolve the full URI automatically; you can see this in action if you use --log-http with your gcloud command:

gcloud beta dataproc clusters create foo --image=custom-image-name --log-http

If you created one with gcloud you can also gcloud dataproc clusters describe your cluster to see the fully-resolved custom image URI.

Upvotes: 1

Related Questions