failed on start-all.sh in Apache Hadoop single node setup

Question

I installed Apache Hadoop 1.0.4 on Ubuntu 12.04. I followed the instruction on http://hadoop.apache.org/docs/stable/single_node_setup.html, and reached the section of "Execution". There I failed on $ bin/start-all.sh, and the error messages are as below. My username is anson.

$ start-all.sh
mkdir: cannot create directory `/var/log/hadoop/anson': Permission denied
chown: cannot access `/var/log/hadoop/anson': No such file or directory
mkdir: cannot create directory `/var/run/hadoop': Permission denied
starting namenode, logging to /var/log/hadoop/anson/hadoop-anson-namenode-anson-Ubuntu-T430.out
/usr/sbin/hadoop-daemon.sh: line 136: /var/run/hadoop/hadoop-anson-namenode.pid: No such file or directory
/usr/sbin/hadoop-daemon.sh: line 135: /var/log/hadoop/anson/hadoop-anson-namenode-anson-Ubuntu-T430.out: No such file or directory
head: cannot open `/var/log/hadoop/anson/hadoop-anson-namenode-anson-Ubuntu-T430.out' for reading: No such file or directory
localhost: mkdir: cannot create directory `/var/log/hadoop/anson': Permission denied
localhost: chown: cannot access `/var/log/hadoop/anson': No such file or directory
localhost: mkdir: cannot create directory `/var/run/hadoop': Permission denied
localhost: starting datanode, logging to /var/log/hadoop/anson/hadoop-anson-datanode-anson-Ubuntu-T430.out
localhost: /usr/sbin/hadoop-daemon.sh: line 136: /var/run/hadoop/hadoop-anson-datanode.pid: No such file or directory
localhost: /usr/sbin/hadoop-daemon.sh: line 135: /var/log/hadoop/anson/hadoop-anson-datanode-anson-Ubuntu-T430.out: No such file or directory
localhost: head: cannot open `/var/log/hadoop/anson/hadoop-anson-datanode-anson-Ubuntu-T430.out' for reading: No such file or directory
localhost: mkdir: cannot create directory `/var/log/hadoop/anson': Permission denied
localhost: chown: cannot access `/var/log/hadoop/anson': No such file or directory
localhost: mkdir: cannot create directory `/var/run/hadoop': Permission denied
localhost: starting secondarynamenode, logging to /var/log/hadoop/anson/hadoop-anson-secondarynamenode-anson-Ubuntu-T430.out
localhost: /usr/sbin/hadoop-daemon.sh: line 136: /var/run/hadoop/hadoop-anson-secondarynamenode.pid: No such file or directory
localhost: /usr/sbin/hadoop-daemon.sh: line 135: /var/log/hadoop/anson/hadoop-anson-secondarynamenode-anson-Ubuntu-T430.out: No such file or directory
localhost: head: cannot open `/var/log/hadoop/anson/hadoop-anson-secondarynamenode-anson-Ubuntu-T430.out' for reading: No such file or directory
mkdir: cannot create directory `/var/log/hadoop/anson': Permission denied
chown: cannot access `/var/log/hadoop/anson': No such file or directory
mkdir: cannot create directory `/var/run/hadoop': Permission denied
starting jobtracker, logging to /var/log/hadoop/anson/hadoop-anson-jobtracker-anson-Ubuntu-T430.out
/usr/sbin/hadoop-daemon.sh: line 136: /var/run/hadoop/hadoop-anson-jobtracker.pid: No such file or directory
/usr/sbin/hadoop-daemon.sh: line 135: /var/log/hadoop/anson/hadoop-anson-jobtracker-anson-Ubuntu-T430.out: No such file or directory
head: cannot open `/var/log/hadoop/anson/hadoop-anson-jobtracker-anson-Ubuntu-T430.out' for reading: No such file or directory
localhost: mkdir: cannot create directory `/var/log/hadoop/anson': Permission denied
localhost: chown: cannot access `/var/log/hadoop/anson': No such file or directory
localhost: mkdir: cannot create directory `/var/run/hadoop': Permission denied
localhost: starting tasktracker, logging to /var/log/hadoop/anson/hadoop-anson-tasktracker-anson-Ubuntu-T430.out
localhost: /usr/sbin/hadoop-daemon.sh: line 136: /var/run/hadoop/hadoop-anson-tasktracker.pid: No such file or directory
localhost: /usr/sbin/hadoop-daemon.sh: line 135: /var/log/hadoop/anson/hadoop-anson-tasktracker-anson-Ubuntu-T430.out: No such file or directory
localhost: head: cannot open `/var/log/hadoop/anson/hadoop-anson-tasktracker-anson-Ubuntu-T430.out' for reading: No such file or directory
anson@anson-Ubuntu-T430:~$ cd /var/run/hadoop
-bash: cd: /var/run/hadoop: No such file or directory

Answer 1

It sounds like you installed the Hadoop Debian package, and you're trying to run start-all.sh as a non-root user.

Here are some ideas:

1. Try following the "Standalone Operation" instructions, that should work as written.
2. Try following the "Pseudo-Distributed Operation" section as written with one exception: sudo start-all.sh instead of bin/start-all.sh.
3. Try completely uninstalling the Debian package and use a Hadoop release tarball instead.

If you go with the tarball option, first purge the Debian package:

sudo apt-get purge hadoop

Make sure you don't have any Hadoop-specific environment variables set. This should return nothing:

set | grep HADOOP

If you get any output from that command, reboot or manually unset any environment variables containing HADOOP.

At this point you should be able to explode the tarball and follow the "Pseudo-Distributed Operation" guide as written (and as a non-root user).

There are two tarballs. hadoop-1.0.4.tar.gz includes source and docs, hadoop-1.0.4-bin.tar.gz does not. Either works.

Answer 2

change the owner of the directory and also permissions to 755

Answer 3

The user you used for start hadoop have no permission on /var/log. Check the user's permissions on /var/log/.

Another way, you can change the hadoop base dir across hadoop.tmp.dir configuration.