CODEWITHSUNDEEP
hadoopresetjobs
Razvan
Razvan

Reputation: 10093

Hadoop reuse Job object

I have a pool of Jobs from which I retrieve jobs and start them. The pattern is something like: 

    Job job = JobPool.getJob();
    job.waitForCompletion();
    JobPool.release(job);

I get a problem when I try to reuse a job object in the sense that it doesn't even run (most probably because it's status is : COMPLETED). So, in the following snippet the second waitForCompletion call prints the statistics/counters of the job and doesn't do anything else. 

    Job jobX = JobPool.getJob();
    jobX.waitForCompletion();
    JobPool.release(jobX);

    //.......

    Job jobX = JobPool.getJob();
    jobX.waitForCompletion(); // <--- here the job should run, but it doesn't

Am I right when I say that the job doesn't actually run because hadoop sees its status as completed and it doesn't event try to run it ? If yes, do you know how to reset a job object so that I can reuse it ?

Upvotes: 0

Views: 279

Answers (1)

VoiceOfUnreason
VoiceOfUnreason

Reputation: 57277

The Javadoc includes this hint that the jobs should only run once

The set methods only work until the job is submitted, afterwards they will throw an IllegalStateException.

I think there's some confusion about the job, and the view of the job. The latter is the thing that you have got, and it is designed to map to at most one job running in hadoop. The view of the job is fundamentally light weight, and if creating that object is expensive relative to actually running the job... well, I've got to believe that your jobs are simple enough that you don't need hadoop.

Using the view to submit a job is potentially expensive (copying jars into the cluster, initializing the job in the JobTracker, and so on); conceptually, the idea of telling the jobtracker to "rerun " or "copy ; run ", makes sense. As far as I can tell, there's no support for either of those ideas in practice. I suspect that hadoop isn't actually guaranteeing retention policies that would support either use case.

Upvotes: 1

Related Questions