CODEWITHSUNDEEP
apache-sparkhadoop-yarn
Tiffany
Tiffany

Reputation: 273

How to access only spark logs in a yarn cluster

I'm using spark with yarn in cluster mode and I wouldlike to retrieve only the logs concerning spark in order to use the history server. Is there a way to do that instead of copy pasting it from the big log files ?

Upvotes: 1

Views: 1478

Answers (1)

Yehor Krivokon
Yehor Krivokon

Reputation: 877

Yes, you can take logs of each task. You need to go to the yarn UI (8088 port) and copy id of the task. Then use command:

yarn logs --applicationId <TASK_ID>

For example: 

yarn logs --applicationId application_1504771895665_0001

By this command you can see logs of each container that task used.

Upvotes: 3

Related Questions