CODEWITHSUNDEEP
hadoopamazon-s3apache-sparkjets3t
Daniel Mahler
Daniel Mahler

Reputation: 8193

error when reading from S3 using Spark/Hadoop

I am trying to read data from Amazon S3 using the Spark. but I am getting

java.lang.NoClassDefFoundError: org/jets3t/service/S3ServiceException

from inside a Hadoop call. I have tried dwnloading jets3t and adding all the included jars to my classpath, but it did not help. Here is the full transcript of what is happening:

scala> val zz = sc.textFile("s3n:/<bucket>/<path>")
13/08/30 19:50:21 INFO storage.MemoryStore: ensureFreeSpace(45979) called with curMem=46019, maxMem=8579469803
13/08/30 19:50:21 INFO storage.MemoryStore: Block broadcast_1 stored as values to memory (estimated size 44.9 KB, free 8.0 GB)
zz: spark.RDD[String] = MappedRDD[3] at textFile at <console>:12

scala> zz.first
13/08/30 19:50:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/08/30 19:50:38 WARN snappy.LoadSnappy: Snappy native library not loaded
java.lang.NoClassDefFoundError: org/jets3t/service/S3ServiceException
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.createDefaultStore(NativeS3FileSystem.java:224)
at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:214)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:176)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
at spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:76)
at spark.RDD.partitions(RDD.scala:214)
at spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:26)
at spark.RDD.partitions(RDD.scala:214)
at spark.RDD.take(RDD.scala:764)
at spark.RDD.first(RDD.scala:778)

Upvotes: 3

Views: 1947

Answers (1)

jeremyjjbrown
jeremyjjbrown

Reputation: 8009

When you run a Hadoop job you have to set the Hadoop classpath Environmental Variable. Typically, this is done in the Hadoop startup script.

export HADOOP_CLASSPATH=/path/to/yourlib:/path/to/your/other/lib

replace the : with a ; if you are on windows.

Upvotes: 1

Related Questions