CODEWITHSUNDEEP
amazon-web-servicesapache-sparksshhadoop-yarnamazon-emr
Alon
Alon

Reputation: 11885

Cannot ssh into Spark worker

There are 8 failed tasks in a particular executor. I want to connect to it via ssh to view the yarn logs.

The executor address is: ip-123-45-6-78.us-west-2.compute.internal:34265

I've tried both:

ssh ip-123-45-6-78.us-west-2.compute.internal:34265

and

ssh ip-123-45-6-78.us-west-2.compute.internal

But both produce the following error:

Could not resolve hostname ip-123-45-6-78.us-west-2.compute.internal: Name or service not known

I've also added to the .ssh/config file the same key-pair I use to connect to the master:

Host master
        HostName ec2-09-876-543-21.us-west-2.compute.amazonaws.com
        User hadoop
        IdentityFile ~/keypair.pem

Host worker
        HostName ip-123-45-6-78.us-west-2.compute.internal
        User hadoop
        IdentityFile ~/keypair.pem

And also both ssh worker and ssh worker:34265 don't work.

Just to be clear: ssh master does work!

The Spark application is running on an EMR cluster.

Upvotes: 0

Views: 910

Answers (2)

Chris Williams
Chris Williams

Reputation: 35146

The hostname you're trying to connect to will not resolve as you're outside of the AWS VPC. Private records (those as part of the compute.internal domain) only resolve if the DNS of the network goes through the Route 53 Private Resolver.

If you're not to worried about resolving the DNS hostnames you can instead attempt connecting via the private IP directly (assuming you have access via either a VPN connection or Direct Connect). Alternatively connect via an instance that has public ingress i.e. Client -> Jump Server -> Private Host.

If you do want to resolve via private domain name the following are the best options:

Upvotes: 0

Adiii
Adiii

Reputation: 59896

From the hostname *.compute.internal these are internal IP address (private IP) and you can not ssh from your local system.

You are able to SSH to master because you are using public IP address of the master instance. try to use the public IP address for the worker too and it should work.

Or the option is to create ssh-tunnel through the master server, you can try something like

Host worker
        HostName ip-123-45-6-78.us-west-2.compute.internal
        User hadoop
        IdentityFile ~/keypair.pem
        ProxyCommand ssh master -W %h:%p

Upvotes: 1

Related Questions