CODEWITHSUNDEEP
unixhadoopmapreducehadoop-yarnuser-permissions
MiamiBeach
MiamiBeach

Reputation: 3497

Hadoop: setting MapReduce resource permissions

Given we have some Hadoop MapReduce task to be run. This MapReduce needs to access some system resources on local drive, i.e. on some node (in fact, we have to place that resources to all nodes).

A question is: which permissions should be given to that resource file?

I would like to give it permissions to be read by the user which runs Hadoop. But in fact the task will be executed under another user. That user is 'yarn'. I.e. if I want to place some resources to some home folder of user which runs Hadoop Job, or related Oozie job etc I cannot do it because in fact home folder of the user which owns MapReduce is /home/yarn/.

What is the best way to deal with this issue? How do I control under which user MapReduce runs? Where can I lookup that settings?

Upvotes: 0

Views: 230

Answers (2)

Prabhu Moorthy
Prabhu Moorthy

Reputation: 177

First off the statement "MapReduce needs to access some system resources on local drive" is not possible when running a MapReduce program in distributed mode. Whatever file you need should be moved to HDFS. Give the file a read permission to all users I hope everything should be fine. If you need to read the file in the Mapper or Reducer and not pass the file as the input to the MapReduce program then consider using Distributed Cache mechanism provided my MapReduce.

Upvotes: 0

Marco99
Marco99

Reputation: 1659

I guess all you need is to create the required folders for such resources in HDFS, and set the permissions to those folders and the contained files using 'hadoop fs -chmod ..' command.

Please refer this below link: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html

Upvotes: 0

Related Questions