CODEWITHSUNDEEP
hadoop
David Phillips
David Phillips

Reputation: 10208

Get the task attempt ID for the currently running Hadoop task

The Task Side-Effect Files section of the Hadoop tutorial mentions using the "attemptid" of the task as a unique name. How do I get this attempt ID in my mapper or reducer?

Upvotes: 8

Views: 8401

Answers (3)

Brian Bloniarz
Brian Bloniarz

Reputation: 411

With the new Hadoop API:

context.getTaskAttemptID().getTaskID().getId()

Upvotes: 10

CalloRico
CalloRico

Reputation: 1158

Late to the party, but you can use the TaskAttemptID class to parse the mapred.task.id property.

In my case, I wanted the numeric attempt value itself and used the following in my Mapper:

int _attemptID;

@Override
public void configure(JobConf conf) {
    TaskAttemptID attempt = TaskAttemptID.forName(conf.get("mapred.task.id"));
    _attemptID = attempt.id();
}

Upvotes: 4

Dain Sundstrom
Dain Sundstrom

Reputation:

If you need a unique id for a side effect file in hadoop, you can leverage the attempt unique id in the job with this code:

   public static String getAttemptId(Configuration conf) throws IllegalArgumentException
   {
       if (conf == null) {
           throw new NullPointerException("conf is null");
       }

       String taskId = conf.get("mapred.task.id");
       if (taskId == null) {
           throw new IllegalArgumentException("Configutaion does not contain the property mapred.task.id");
       }

       String[] parts = taskId.split("_");
       if (parts.length != 6 ||
               !parts[0].equals("attempt") ||
               (!"m".equals(parts[3]) && !"r".equals(parts[3]))) {
           throw new IllegalArgumentException("TaskAttemptId string : " + taskId + " is not properly formed");
       }

       return parts[4] + "-" + parts[5];
   }

Upvotes: 11

Related Questions