CODEWITHSUNDEEP
hadoophadoop-yarn
quasiben
quasiben

Reputation: 1464

YARN ContainerIDs from Client

Is it possible to get the various container IDs and hostnames from the YARN client after the client has submitted the appContext ? I suppose that would mean after the containers have been allocated can the information be communicated back to the client?

If this is not possible does the RM Web Interface provide this info ?

Upvotes: 4

Views: 1718

Answers (2)

Manjunath Ballur
Manjunath Ballur

Reputation: 6343

YARN CLI has an option to get the list of containers from the Application Attempt ID. You can get the list of containers, by following the steps below.

  • Step 1: Get the list of Application Attempts from the Application ID

    From the YARN application ID, you can get a list of application attempts, using the following command:

    yarn applicationattempt -list <Application ID>

    The description of this command reads:

    -list <Application ID>    List application attempts for application.

    For e.g.

    yarn applicationattempt -list application_1452267331813_0009

  • Step 2: Get the containers using Application Attempt ID

    After you get list of application attempts, for each application attempt, you can get a list of containers, using following command:

    yarn container -list <Application Attempt ID>

    The description of this command reads:

    -list <Application Attempt ID>    List containers for application attempt.

    For e.g.: 

    yarn container -list appattempt_1452267331813_0009_000001

Upvotes: 5

quasiben
quasiben

Reputation: 1464

Rather ugly but can be done through a series URL requests to the ResourceManager, NodeManager, and various ContainerLog urls:

# Resource Manager Info for Given Application
curl http://RM:PORT/ws/v1/cluster/apps/<app_id>

amHostHttpAddress = data['app']['amHostHttpAddress']

# List containers with id, nodeId, containerLogsLink, etc
curl http://<amHostHttpAddress>/ws/v1/node/containers

# stdout/stderr for a specific container
curl http://<containerLogsLink>/stdout/?start=0"
curl http://<containerLogsLink>/stderr/?start=0"

This method works but there should probably be a cleaner way of doing this in YARN JAVA API

Upvotes: 0

Related Questions