CODEWITHSUNDEEP
javajakarta-eehadoopmapreducehdfs
yzandrew
yzandrew

Reputation: 785

How can I run a mapreduce job remotely

I have met some issues when I was trying to run a map-reduce job (word-count example) remotely. After I search on google I still can't achieve my goal. And I just saw very few topics about invoking the map-reduce job remotely. Below is the issues:

  1. At first, I meet permission issue:

    SEVERE: PriviledgedActionException as:[user]    cause:org.apache.hadoop.security.AccessControlException:
org.apache.hadoop.security.AccessControlException: Permission denied: user=[user], access=WRITE, inode="mapred":root:supergroup:rwxr-xr-x

    It seems like a permission deny to hdfs path. I turn off the checking by setting dfs.permissions = true. Is there other way to overcome this problem but still keep the checking on.

  2. Then I met a exception saying I can't access the map-reduce application jar. 

    SEVERE: PriviledgedActionException as:[User] cause:java.io.FileNotFoundException: File /home/hduser/WordCount-1.0.jar does not exist.

    My code is:

    JobConf conf = new JobConf(WordCount.class);
conf.setJobName("wordcount");
conf.set("fs.default.name", "hdfs://10.97.1.91:54310");
conf.set("mapred.job.tracker", "10.97.1.91:54311");
conf.setJar("/home/hduser/WordCount-1.0.jar");
...
JobClient.runJob(conf);

    It seems like the local file system on name node deny my access to the jar file. How can I overcome this issue? I found a link in stackoverflow. From the link, the jar file location is not necessary. But how can the name node find the jar file without providing the location?

  3. I saw very few example executing a job remotely in the tutorial I found in internet. Is it not suggest to do it this way?

Upvotes: 4

Views: 2163

Answers (1)

Magham Ravi
Magham Ravi

Reputation: 603

To the first issue, looks like [user] doesn't have permissions on HDFS. Primarily, Hadoop uses whoami command to determine if the user submitting the job and the output from the command is same. Two ways to address this, a) Determine the user who has permissions on HDFS, say root, and add the following line before submitting your job. System.setProperty("HADOOP_USER_NAME","root"); b) Impersonation. Check out the following link http://hadoop.apache.org/docs/stable/Secure_Impersonation.html

Upvotes: 1

Related Questions