CODEWITHSUNDEEP
hadoopamazon-ec2emr
user1912731
user1912731

Reputation: 11

Access EMR Task logs from Hadoop Job Tracker web interface

I've opened the EC2 security groups for my master/slave nodes so I can access the job tracker interface from my local browser. I connect to it using http://MASTER-IP:9100.

Everything works well until I try to access the task tracker logs from the task details - http://ec2-xx-xx-xx-xx.compute-1.amazonaws.com:9100/taskdetails.jsp?tipid=task_201212181113_0001_m_000000

The links I get there are to the internal EC2 IP addresses, therefore I cannot access them from my local machine (links are http://10.116.xxx.xx:9103/tasklog?attemptid=attempt_201212181113_0001_m_000000_0&all=true )

Is there a way to change EMR so that those links will contain the task tracker's public IP?

Thanks!

Upvotes: 1

Views: 3336

Answers (3)

user373480
user373480

Reputation: 173

Try ssh tunneling. And use Firefox-Plugin FoxyProxy. you should be able to see the job tracker interface from the local browser and browse the logs

Upvotes: 0

Amar
Amar

Reputation: 12010

What you can do very easily is ssh to the master node and from there you cab to a curl to this log location. Generally we always tunnel to the master node and access the Job tracker web iterface via http://localhost:9100

In order to tunnel you can do the following:

ssh -i your-pem-file.pem -L 9100:ec2-xx-xx-xx-xx.computamazonaws.com:9100 [email protected]

And then access the logs via simple curl, something like :

$ curl http://10.116.xxx.xx:9103/tasklog?attemptid=attempt_201212181113_0001_m_000000_0&all=true

You can also login to your aws console and there look for the list of running ec2 instances. From there you can get the private IP and corresponding public IP for the machine's log you want, you can then replace the private IP with this retrieved public IP in the log's URL and you are good to go.

But first solution is pretty simple and is what we do generally.

Upvotes: 4

Tariq
Tariq

Reputation: 34184

Add the public IPs of all the slaves along with their Hostnames into your /etc/hosts file.

Upvotes: 2

Related Questions