CODEWITHSUNDEEP
scalaapache-sparksequencefile
zqiang0zh
zqiang0zh

Reputation: 91

Two ways to read hdfs sequece file, but one failed with EOFException

Use sparkContext to read sequence file, as follows:

Method 1:

val rdd = sc.sequenceFile(path, classOf[BytesWritable], 
          classOf[BytesWritable])
rdd.count()

Method 2:

val rdd = sc.hadoopFile(path,
          classOf[SequenceFileAsBinaryInputFormat],
          classOf[BytesWritable],
          classOf[BytesWritable])
rdd.count()

Method 1 end up with EOFException, but method 2 works. What's the differences of these two methods?

Upvotes: 0

Views: 298

Answers (1)

uh_big_mike_boi
uh_big_mike_boi

Reputation: 3470

The difference starts where "Method 1" immediately makes the call hadoopFile(path, inputFormatClass, keyClass, valueClass, minPartitions) which uses SequenceFileInputFormat[BytesWritable, BytesWritable], but "Method 2" makes the same call except of course uses SequenceFileAsBinaryInputFormat.
Then to continue, even though SequenceFileAsBinaryInputFormat extends SequenceFileInputFormat[BytesWritable, BytesWritable], SequenceFileAsBinaryInputFormat has it's own inner class called SequenceFileAsBinaryRecordReader and although it works similarly to SequenceFileRecordReader[BytesWritable, BytesWritable], there are differences. When we take a look at the code, they are doing some different implementations, namely the former is handling compression better. So if your sequence file is either record compressed or block compressed then it makes sense that SequenceFileInputFormat[BytesWritable, BytesWritable] is not iterating with the same dependability as SequenceFileAsBinaryInputFormat.

SequenceFileAsBinaryInputFormat which uses SequenceFileAsBinaryRecordReader (lines 102-115) - https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SequenceFileAsBinaryInputFormat.java

SequenceFileRecordReader (lines 79 - 91) - https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SequenceFileRecordReader.java

Upvotes: 2

Related Questions