CODEWITHSUNDEEP
hadoopmapreduce
Ilion
Ilion

Reputation: 6882

Does it matter where I submit hadoop jobs from?

Does it have any measurable effect on resources whether I submit a bunch of hadoop jobs from different client servers or all from the same one? I would think not since all the work is done in the cluster. Is this correct?

Upvotes: 3

Views: 430

Answers (3)

Arun J
Arun J

Reputation: 557

I don't think you can move the input split calculation into Job Tracker in 'Classic' version. In YARN, you can move it using

"yarn.app.mapreduce.am.compute-splits-in-cluster"

I am guessing, Hadoop people didn't want to overload Job tracker with input split creation. Similar to the design decision of not assigning too much work for Namenode in HDFS.

In YARN, every job gets its own Application Master, so no worries about overloading a SPOF/bottleneck master like job tracker.

In reference to the original question, the client job would have to reach out to the namenode to get the block locations (I have see parts of code on block storage class calling data node for some meta data...not sure whether these happen during input split creation or in task tracker node) . This can become an issue if you are handling a lot of jobs on the same client node. If you are using YARN, there would be a slight performance increase if all these communications happen inside the cluster.

Need to check how Oozie handles this issue.

Hopefully, this helps! Arun

Upvotes: 1

Praveen Sripati
Praveen Sripati

Reputation: 33545

The only thing which is resource intensive on the client submitting to the Hadoop cluster is the calculation of the input splits. When the input data is huge or when too many jobs are submitted from the same client then because of the input split calculations, the job submission might become a bit slow.

I am not able to recall the Hadoop release or the parameter, but a configurable parameter was included to move the calculation of the input splits from the client submitting a job to the Hadoop cluster.

Upvotes: 6

Charles Menguy
Charles Menguy

Reputation: 41458

It really shouldn't matter where you submit your jobs from. The client itself doesn't do much, it uses RPC protocol to contact the services, and then just sits idle until the job is finished.

Also, the most important is what kind of scheduler you use to allocate resource, which is probably going to make the most significant difference and decide which resources to allocate to which job. More on job scheduling here.

Upvotes: 3

Related Questions