CODEWITHSUNDEEP
javaeclipsehadoopcygwin
user3260156
user3260156

Reputation:

Invalid directory in dfs.data.dir: Failed to set permissions of path: \tmp\hadoop-user\dfs\data to 0755

i am new to hadoop framework , currently i am working with big data projects, using cygwin , hadoop-0.19.1 ,eclipse-3.3.1 (Europa) in windows 7. now i am trying to change from hadoop -0.19.1 to hadoop-1.2.1 version.i configured hadoop-1.2.1 as follows

core-site.xml:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

 <property>
         <name>fs.default.name</name>
         <value>hdfs://localhost:9100</value>
  </property>

</configuration>

hdfs.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

  <property>
         <name>dfs.replication</name>
         <value>1</value>
     </property>


</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

     <property>
         <name>mapred.job.tracker</name>
         <value>localhost:9101</value>
     </property>



</configuration>

but i got error in starting datanode as below 

$ bin/hadoop datanode
14/02/07 22:24:09 INFO datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = raj-PC/101.63.181.80
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/b                                                                                                                ranch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.7.0_45
************************************************************/
14/02/07 22:24:10 INFO impl.MetricsConfig: loaded properties from hadoop-metrics                                                                                                                2.properties
14/02/07 22:24:10 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem                                                                                                                ,sub=Stats registered.
14/02/07 22:24:10 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 s                                                                                                                econd(s).
14/02/07 22:24:10 INFO impl.MetricsSystemImpl: DataNode metrics system started
14/02/07 22:24:11 INFO impl.MetricsSourceAdapter: MBean for source ugi registere                                                                                                                d.
14/02/07 22:24:11 WARN impl.MetricsSystemImpl: Source name ugi already exists!
14/02/07 22:24:11 WARN util.NativeCodeLoader: Unable to load native-hadoop libra                                                                                                                ry for your platform... using builtin-java classes where applicable
14/02/07 22:24:11 WARN datanode.DataNode: Invalid directory in dfs.data.dir: Fai                                                                                                                led to set permissions of path: \tmp\hadoop-raj\dfs\data to 0755
14/02/07 22:24:11 ERROR datanode.DataNode: All directories in dfs.data.dir are i                                                                                                                nvalid.
14/02/07 22:24:11 INFO datanode.DataNode: Exiting Datanode
14/02/07 22:24:11 INFO datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at raj-PC/101.63.181.80
************************************************************/

. i tried patch for windows from this github site: https://github.com/congainc/patch-hadoop_7682-1.0.x-win.

it solves my problem. but if i tried to run simple word count program in eclipse it gives me same error there:"Failed to set permissions of path: \tmp\hadoop-user\dfs\data to 0755 ".

i have googled many times but didnt get any solution.

any body pls help.

After Changing the hdfs-site.xml as told.

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

  <property>
         <name>dfs.replication</name>
         <value>1</value>
     </property>

  <property>
    <name>dfs.name.dir</name>
    <value>C:/cygwin/usr/tmp/hadoop-raj/dfs/logs</value>
 </property>

  <property>
    <name>dfs.data.dir</name>
    <value>c:/cygwin/usr/hadoop-raj/dfs/data</value>
 </property>


  <property>
<name>hadoop.tmp.dir</name>
<value>c:/cygwin/usr/hadoop-raj/dfs/tmp</value>
</property>


</configuration>

Now i am getting following error at tasktracker

 $ bin/hadoop tasktracker
14/02/08 15:25:22 INFO mapred.TaskTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG:   host = raj-PC/192.168.42.248
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 1.2.1
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/b                                                                                                                ranch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013
STARTUP_MSG:   java = 1.7.0_45
************************************************************/
14/02/08 15:25:23 INFO impl.MetricsConfig: loaded properties from hadoop-metrics                                                                                                                2.properties
14/02/08 15:25:23 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem                                                                                                                ,sub=Stats registered.
14/02/08 15:25:23 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 s                                                                                                                econd(s).
14/02/08 15:25:23 INFO impl.MetricsSystemImpl: TaskTracker metrics system starte                                                                                                                d
14/02/08 15:25:23 WARN util.NativeCodeLoader: Unable to load native-hadoop libra                                                                                                                ry for your platform... using builtin-java classes where applicable
14/02/08 15:25:23 INFO impl.MetricsSourceAdapter: MBean for source ugi registere                                                                                                                d.
14/02/08 15:25:23 WARN impl.MetricsSystemImpl: Source name ugi already exists!
14/02/08 15:25:23 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter                                                                                                                (org.mortbay.log) via org.mortbay.log.Slf4jLog
14/02/08 15:25:23 INFO http.HttpServer: Added global filtersafety (class=org.apa                                                                                                                che.hadoop.http.HttpServer$QuotingInputFilter)
14/02/08 15:25:23 INFO mapred.TaskLogsTruncater: Initializing logs' truncater wi                                                                                                                th mapRetainSize=-1 and reduceRetainSize=-1
14/02/08 15:25:23 INFO mapred.TaskTracker: Starting tasktracker with owner as ra                                                                                                                j
14/02/08 15:25:23 INFO mapred.TaskTracker: Good mapred local directories are: /t                                                                                                                mp/hadoop-raj/mapred/local
14/02/08 15:25:23 ERROR mapred.TaskTracker: Can not start task tracker because j                                                                                                                ava.io.IOException: Failed to set permissions of path: \tmp\hadoop-raj\mapred\lo                                                                                                                cal\taskTracker to 0755
        at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:691)
        at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:672)
        at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSys                                                                                                                tem.java:514)
        at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.jav                                                                                                                a:349)
        at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:19                                                                                                                3)
        at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:823)
        at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1573)
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3937)

14/02/08 15:25:23 INFO mapred.TaskTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down TaskTracker at raj-PC/192.168.42.248
************************************************************/

Upvotes: 2

Views: 1851

Answers (1)

LeonardBlunderbuss
LeonardBlunderbuss

Reputation: 1274

This is a Cygwin issue. You need to change all local directories to Cygwin directories, such as c:/cygwin/usr/tmp/hadoop-user/dfs/data

Make sure you do this with your temp, logs, and data directories (hdfs-site.xml)

Upvotes: 1

Related Questions