CODEWITHSUNDEEP
hadoop
user1050619
user1050619

Reputation: 20916

Redirect Hadoop job output to a file

Im running a Hadoop job and outputs are displayed on the console. Is there a way for me to redirect the output to a file..I tried the below command to redirect the output but it does not work.

hduser@vagrant:/usr/local/hadoop$ hadoop jar share/hadoop/mapreduce/hadoop*examples*.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output>joboutput

Upvotes: 1

Views: 3539

Answers (2)

Ziya Karakaya
Ziya Karakaya

Reputation: 11

You can redirect the error stream to file, which is the output of hadoop job. That is use;

hadoop jar share/hadoop/mapreduce/hadoop*examples*.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output 2>joboutput

Upvotes: 1

vlahmot
vlahmot

Reputation: 116

If you are running the examples from the Hadoop homepage (https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html) the output will be written to

/user/hduser/gutenberg /user/hduser/gutenberg-output

on HDFS and not the local file system.

You can see the output via

hadoop fs -text /user/hduser/gutenberg /user/hduser/gutenberg-output/*

And to dump that output to a local file 

hadoop fs -text /user/hduser/gutenberg /user/hduser/gutenberg-output/* > local.txt

The -text option will decompress the data so you get textual output in case you have some type of compression enabled.

Upvotes: 0

Related Questions