CODEWITHSUNDEEP
javaamazon-web-servicesapache-sparkemr
Shahar Cohen
Shahar Cohen

Reputation: 41

How to configure Java client connecting to AWS EMR spark cluster

I'm trying to write a simple spark application, and when i run it locally it works with setting the master as

.master("local[2]")

But after configuring spark cluster on AWS (EMR) i can't connet to the master url:

.master("spark://<master url>:7077")

Is this the way to do it? am i missing something here? The cluster is up and running, and when i tried adding my application as a step jar, so it will run directly in the cluster it worked. But i want to be able to run it from a remote machine.

would appreciate some help here, Thanks

Upvotes: 4

Views: 547

Answers (1)

Ryan Widmaier
Ryan Widmaier

Reputation: 8513

To run from a remote machine, you will need to open the appropriate ports in the Security Group assigned to your EMR master node. You will need to add at least 7077.

If by "remote" you mean one that isn't in your AWS environment, you will also need to setup a way to route traffic to it from the outside.

Upvotes: 1

Related Questions