CODEWITHSUNDEEP
hadoopapache-sparkhdfs
Jack
Jack

Reputation: 5870

spark1.3 can not read data from HDFS1

I'm using Spark1.3.1 trying to read data from HDFS as:

val sc = new SparkContext(sparkConf)
val lines = sc.textFile("hdfs://192.168.0.104:9000/cur/part-r-02554")

I met the following excepton:

Exception in thread "main" java.io.IOException: Failed on local exception: 
com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group
tag did not match expected tag.; Host Details : local host is: 
"hadoop104/192.1168.1.104"; destination host is: "hadoop104":9000;

Upvotes: 0

Views: 734

Answers (1)

Harman
Harman

Reputation: 727

Try looking for property fs.defaultFS or fs.default.name in your core-site.xml. Check if 192.168.0.104 is configured as value, and not a hostname.

If a hostname is configured as a value, this is bound to give you an error - as this is very strictly followed. Either, use whatever is configured in core-site.xml or do not use an IP/hostname and just go ahead with hdfs:/cur/part-r-02554

Upvotes: 1

Related Questions