CODEWITHSUNDEEP
scalaapache-sparkgoogle-cloud-platformgoogle-cloud-dataproc
BAR
BAR

Reputation: 17071

Google Cloud Dataproc - Submit Spark Jobs Via Spark

Is there a way to submit Spark jobs to Google Cloud Dataproc from within the Scala code?

  val Config = new SparkConf()
    .setMaster("...")

What should the master URI look like?

What key-value pairs should be set to authenticate with an API key or keypair?

Upvotes: 2

Views: 969

Answers (1)

James
James

Reputation: 2331

In this case, I'd strongly recommend an alternative approach. This type of connectivity has not been tested or recommended for a few reasons:

  1. It requires opening firewall ports to connect to the cluster
  2. Unless you use a tunnel, your data may be exposed
  3. Authentication is not enabled by default

Is SSHing into the master node (the node which is named cluster-name-m) a non-starter? It is pretty easy to SSH into the master node to directly use Spark.

Upvotes: 1

Related Questions