HADOOP - Writing YARN applications

Question

I am reading this and I am a little bit confused. Do I have to write always Client and ApplicationMaster Class to run my own code in YARN? For example if I want to run an application which just generates random numbers, Do I have to include these classes to my code or it can run it without these? Or, are they both different examples because I can see 2 main classes in there (1 main in each class)? I tried it with all of those ways but I am still getting something like an error --> usage [input][ouput]:

Answer 1

You could use provided examples in jar file located at $Yarn_Home/share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.1.jar.

example:

yarn jar $Yarn_Home/share/hadoop/mapreduce/hadoop-mapreduce-examples-0.23.1.jar wordcount /in /out

/in and /out are located in hdfs. You need to put some text files in /in and /out must not exist before.

Check this for reference: http://hortonworks.com/blog/introducing-apache-hadoop-yarn/

https://github.com/hortonworks/simple-yarn-app

Answer 2

Writing a native YARN apps may be a little awkward because Hadoop YARN project doesn't have a higher level framework to help with this. However there are few projects which are doing the heavy lifting so that user would not have a need to worry about this boilerplate code.

• Spring YARN as part of Spring for Apache Hadoop with YARN Samples

• Apache Twill

• Slider

  In a link you posted, yes there are two main classes, one for running the client which deploys and starts the app on YARN, other for appmaster.