CODEWITHSUNDEEP
hadoopmapreduce
l a s
l a s

Reputation: 3923

where does hadoop store the output files of mapper, partitioner and combiner?

I am running mapreduce jobs on a pseudo distributed hadoop setup. Where do i find the output file of mapper, partitioner and combiner? Is there a way to inspect the output of each operations?

Upvotes: 1

Views: 433

Answers (1)

Donald Miner
Donald Miner

Reputation: 39943

Intermediate output in MapReduce is stored in local temp storage on the node in which the task ran (not in HDFS).

You can look up in you Hadoop conf where the local temp directories are and go manually inspect them node-by-node.

In general, there might be better ways of doing what you think you want to be doing through log messages or counters. The other thing you can do is turn off reducers so that your Mappers write directly to HDFS so you can inspect that.

Upvotes: 1

Related Questions