CODEWITHSUNDEEP
hadoopapache-sparkhadoop-yarnanalytics-for-apache-hdp
klee
klee

Reputation: 1624

Unable to view logs for yarn-resource-manager

I tried with various command and I still do have issues in viewing the yarn logs and here are the steps that I followed to view logs 

[root@iop-rm1 /]# yarn application -list

17/04/03 13:15:21 INFO client.RMProxy: Connecting to ResourceManager at iop-rm1.qatc/172.18.0.4:8032

Total number of applications (application-types: [] and states: [SUBMITTED, ACCEPTED, RUNNING]):1

            Application-Id      Application-Name        Application-Type          User       Queue               State         Final-State         Progress                        Tracking-URL
  application_1491225022815_0002      paxi                 SPARK                 paxata     default              ACCEPTED         UNDEFINED              0%                                 N/A

When I view the yarn logs it displays as

[root@iop-rm1 /]# yarn logs -applicationId application_1491222823231_0001

17/04/03 12:48:57 INFO client.RMProxy: Connecting to ResourceManager at iop-rm1.qatc/172.18.0.4:8032

/tmp/logs/root/logs/application_1491222823231_0001 does not exist.

Log aggregation has not completed or is not enabled.

I also enabled the Log aggregation in the core-site.xml file

<name>yarn.log-aggregation-enable</name>

<value>true</value>

I created the path for logs which does not exist but i am not able to see yarn logs

Is there any possible way of viewing the yarn logs ?

Upvotes: 2

Views: 6727

Answers (2)

franklinsijo
franklinsijo

Reputation: 18270

You cannot view the aggregated logs of an YARN application which is not yet FINISHED.

From the description of yarn.log-aggregation-enable property:

Log aggregation collects each container's logs and moves these logs onto the directory configured in yarn.nodemanager.remote-app-log-dir only after the completion of the application.

You can try the same command yarn logs -applicationId <application ID> to view the logs once the application has completed.

To view the logs while the job is RUNNING, use the ResourceManger Web Interface. It will be available in http://ResourceManager_IP_Address:8088/.

PS: You should add the property for log aggregation in yarn-site.xml.

Upvotes: 2

ImbaBalboa
ImbaBalboa

Reputation: 867

Maybe you forgot to restart your cluster in order to make the log aggregation enabled.

Upvotes: 0

Related Questions