CODEWITHSUNDEEP
javahadoop
jdevelop
jdevelop

Reputation: 12296

Submitting a job to Hadoop without Hadoop runtime on client

In many manuals there are instructions to submit a job to Hadoop with using of hadoop shellscript, found in Hadoop distribution. As I can see, this implies installation of the entire Hadoop ecosystem on client host, with configuring master node in site settings etc.

Are there any other ways to submit a job without having Hadoop installed on client?

Upvotes: 1

Views: 640

Answers (1)

Tariq
Tariq

Reputation: 34184

Hadoop doesn't stop us from submitting our job from a remote client without having to install Hadoop on it. Any machine sharing the same network can be used to submit a job. A client has nothing more to do than submitting the job and then wait until the job is finished.

That said, there can be several ways to do that. One way could be to run the code directly through your IDE, Eclipse for example. But, you need to point your IDE to the cluster,

conf.set("fs.default.name", "hdfs://NN_HOST:9000");
conf.set("mapred.job.tracker", "JT_HOST:9001");

Another way would be to create a fat jar with all the required binaries. Copy the jar to your JT machine and run it there.

You can find more here and here.

Upvotes: 3

Related Questions