CODEWITHSUNDEEP
apache-sparkhadoop-yarncloudera-cdh
abelard2008
abelard2008

Reputation: 2084

where to check my yarn+spark's applications' log?

I wrote an application with yarn+spark, for simplicity, I list the following 

object testKafkaSparkStreaming extends Logging {

     private class Parser extends Logging{
       def parse(row: String): Row =
       {
         val row = "/_dc.gif| |20160616063934| |39.190.5.69| |729252016040907094857083a3c7c62e"

         logInfo("pengcz starting parse " + row)
       }
     }

    def main(args: Array[String]) {
       ...
       logInfo("main starting parse " + row)
       ...
    }
}

When I executed: 

spark-submit --master local[*] --class $CLASS $JAR ...

I can see the two log infos in the console

But when I executed:

spark-submit --master yarn --class $CLASS $JAR ...

And I opened the yarn web ui of my own application:

http://192.168.36.172:8088/cluster/app/application_1465894400511_3624 enter image description here And I click logs under the page, I got: enter image description here But the page has not information including my two logs

What should I do to find my own logs? Any advice will be appreciated!

Upvotes: 1

Views: 412

Answers (1)

Murtaza Kanchwala
Murtaza Kanchwala

Reputation: 2483

You can use this command to see the Yarn Logs of an Application : 

yarn logs -applicationId <Your Yarn Application ID> // i.e. application_1465894400511_3624

Upvotes: 1

Related Questions