CODEWITHSUNDEEP
hadoopmapreduce
Mahalakshmi Lakshminarayanan
Mahalakshmi Lakshminarayanan

Reputation: 714

Distributed Cache Hadoop not retrieving the file content

I am getting some garbage like value instead of the data from the file I want to use as distributed cache.

The Job Configuration is as follows:

Configuration config5 = new Configuration();
JobConf conf5 = new JobConf(config5, Job5.class);
conf5.setJobName("Job5");
conf5.setOutputKeyClass(Text.class);
conf5.setOutputValueClass(Text.class);
conf5.setMapperClass(MapThree4c.class);
conf5.setReducerClass(ReduceThree5.class);
conf5.setInputFormat(TextInputFormat.class);
conf5.setOutputFormat(TextOutputFormat.class);


DistributedCache.addCacheFile(new URI("/home/users/mlakshm/ap1228"), conf5);
FileInputFormat.setInputPaths(conf5, new Path(other_args.get(5)));
FileOutputFormat.setOutputPath(conf5, new Path(other_args.get(6)));

JobClient.runJob(conf5);

In the Mapper, I have the following code:

public class MapThree4c extends MapReduceBase implements Mapper<LongWritable, Text, 
Text, Text >{
private Set<String> prefixCandidates = new HashSet<String>();

Text a = new Text();
public void configure(JobConf conf5) {

Path[] dates = new Path[0];
try {
        dates = DistributedCache.getLocalCacheFiles(conf5);
        System.out.println("candidates: "+candidates);
        String astr = dates.toString();
        a = new Text(astr);

      } catch (IOException ioe) {
        System.err.println("Caught exception while getting cached files: " +   
      StringUtils.stringifyException(ioe));
      }


  }




   public void map(LongWritable key, Text value, OutputCollector<Text, Text> output, 
   Reporter reporter) throws IOException {

     String line = value.toString();
     StringTokenizer st = new StringTokenizer(line);
     st.nextToken();
     String t = st.nextToken();
     String uidi = st.nextToken();
     String uidj = st.nextToken();

     String check = null;

     output.collect(new Text(line), a);



        }


    }

The output value, I am getting from this mapper is:[Lorg.apache.hadoop.fs.Path;@786c1a82
instead of the value from the distributed cache file.

Upvotes: 0

Views: 1322

Answers (2)

Maha
Maha

Reputation: 173

From your code:

Path[] dates = DistributedCache.getLocalCacheFiles(conf5);

Implies that:

String astr = dates.toString(); // is a pointer to the above array (ie.dates) which is what you see in the output as [Lorg.apache.hadoop.fs.Path;@786c1a82.

You need to do the following to see the actual paths:

for(Path cacheFile: dates){

    output.collect(new Text(line), new Text(cacheFile.getName()));

}

Upvotes: 0

Paul M
Paul M

Reputation: 2046

That looks like what you get when you call toString() on an array and if you look at the javadocs for DistributedCache.getLocalCacheFiles(), that is what it returns. If you need to actually read the contents of the files in the cache, you can open/read them with the standard java APIs.

Upvotes: 1

Related Questions