CODEWITHSUNDEEP
javahadoophortonworks-data-platform
Mittenchops
Mittenchops

Reputation: 19654

Streaming Hadoop jar file in Hortonworks sandbox, no contrib directory

I'm demoing hadoop in the hortonworks sandbox virtual machine.

I've previously set up simple elastic map reduce streaming jobs on EC2, following patterns like this, or this.

However, it seems I have no streaming jar installed---in fact, I seem to be lacking many of the basic directories required in:

$HADOOP_HOME/mapred/contrib/

My ls -lah actually looks like this:

[root@sandbox ~]# ls -lah
total 60K
dr-xr-x---.  5 root root 4.0K Apr 10 18:52 .
dr-xr-xr-x. 24 root root 4.0K Apr 10 18:31 ..
-rw-------   1 root root  126 Oct 28 08:35 .bash_history
-rw-r--r--.  1 root root   18 May 20  2009 .bash_logout
-rw-r--r--.  1 root root  176 May 20  2009 .bash_profile
-rw-r--r--   1 root root  262 Oct 28 08:29 .bashrc
-rw-r--r--.  1 root root  100 Sep 22  2004 .cshrc
-rw-r--r--   1 root root    0 Oct 28 08:34 .hdfs_prepared
drwxr-xr-x   2 root root 4.0K Apr 10 18:22 .pip
drwxr-----   3 root root 4.0K Oct 20 16:21 .pki
-rw-------   1 root root 1.0K Oct 20 14:04 .rnd
drwx------   2 root root 4.0K Oct 20 09:21 .ssh
lrwxrwxrwx   1 root root   48 Oct 28 08:28 start_ambari.sh -> /usr/lib/hue/tools/start_scripts/start_ambari.sh
lrwxrwxrwx   1 root root   47 Oct 28 08:28 start_hbase.sh -> /usr/lib/hue/tools/start_scripts/start_hbase.sh
-rw-r--r--.  1 root root  129 Dec  3  2004 .tcshrc
-rw-------   1 root root 4.8K Oct 28 08:30 .viminfo
-rw-r--r--   1 root root  218 Oct 20 08:55 zero_machine.sh

Although using the hadoop commands, I see that the mapred exists, but contains no contrib.

[root@sandbox ~]# hadoop fs -ls /
Found 6 items
drwxrwxrwt   - yarn   hadoop          0 2014-04-10 19:14 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2013-10-20 15:08 /apps
drwxr-xr-x   - mapred hdfs            0 2013-10-20 15:10 /mapred
drwxr-xr-x   - hdfs   hdfs            0 2013-10-20 15:10 /mr-history
drwxrwxrwx   - hdfs   hdfs            0 2013-10-28 08:34 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2013-10-28 08:34 /user
[root@sandbox ~]# hadoop fs -ls /mapred/
Found 1 items
drwxr-xr-x   - mapred hdfs          0 2013-10-20 15:10 /mapred/system

Is there a special download page for the streaming .jar file? When I access the link at the bottom of this page, the link to streaming is dead.

Upvotes: 2

Views: 3038

Answers (2)

user7610
user7610

Reputation: 28751

There is a convenience link /usr/lib/hadoop-mapreduce/hadoop-streaming.jar to that jar. I tested it in Hortonworks Sandbox 2.1

#ls -AlF /usr/lib/hadoop-mapreduce/hadoop-streaming.jar
lrwxrwxrwx 1 root root 38 Apr 21 06:57 /usr/lib/hadoop-mapreduce/hadoop-streaming.jar -> hadoop-streaming-2.4.0.2.1.1.0-385.jar

Upvotes: 0

SachinJose
SachinJose

Reputation: 8522

The default location of hadoop streaming jar is /usr/lib/hadoop/contrib/streaming/hadoop-streaming-*.jar.

If you cannot find the jar in the above mentioned location. you could download hadoop-streaming-*.jar from the following Hortonworks repository:

http://repo.hortonworks.com/content/repositories/releases/org/apache/hadoop/hadoop-streaming/

Upvotes: 3

Related Questions