CODEWITHSUNDEEP
scalaapache-sparkintellij-ideaamazon-ec2
Siodh
Siodh

Reputation: 136

Trying to do remote debugging an Spark app: java.io.IOException "handshake failed - connection prematurally closed

I'm trying to do remote debugging of a spark app over an AWS EC2 machine.

I have set up an ssh tunnel using:

ssh -i $HOME/pem/dev/dna-dev.pem -N -D 5005 [email protected]

Then added the proper parameters to my spark-submit command:

spark-submit --driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 --class org.sergio.dna.poc.TestEngine   --master yarn   --deploy-mode client   --executor-memory 2G   --num-executors 1 --executor-cores 1  DNA-Sergio-PoC-1.0-SNAPSHOT.jar

And last create a remote debug configuration in Intellij over port 5005

The spark-submit command says its waiting

Listening for transport dt_socket at address: 5005

But when I run the remote debug, Intellij says:

Error running 'RemoteTest': Unable to open debugger port (localhost:5005): java.io.IOException "handshake failed - connection prematurally closed"

Is there any configuration I'm missing? I have done this in cloudera clusters and it worked fine but I'm new in AWS, I wonder if I'm missing something in the server side

Thanks

Upvotes: 2

Views: 551

Answers (1)

Siodh
Siodh

Reputation: 136

At the end I didn't need the ssh tunnel. Just open a port to "All Traffic" in the EMR Console -> Security Group -> master-cluster -> Inbound, 5005 in my case.

Then in your intellij remote debug configuration change "localhost" with your EMR IP.

And it's done.

Upvotes: 2

Related Questions