CODEWITHSUNDEEP
pythonapache-sparkpysparkspark-streamingamazon-kinesis
Lior Baber
Lior Baber

Reputation: 882

pyspark streaming 1.5.0 with kinesis jar missing

Im using EMR (with EMR-4.1.0) including spark 1.5.0 distribution

I tried to use spark streaming (python) to consume data from kinesis using the sample code in github (https://github.com/apache/spark/blob/master/extras/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py)

for some reason, I get an error in which the spark streaming kinesis jar is not available even though I can see it in /usr/lib/spark/extras/lib with all other streaming jars. (see attached file )

-----------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-2-477c4a5455a1> in <module>()
     86     regionName= 'eu-west-1'
     87     lines = KinesisUtils.createStream(
---> 88         ssc, appName, streamName, endpointUrl, regionName, InitialPositionInStream.TRIM_HORIZON, 2)
     89 
     90     words.foreachRDD(process)

/usr/lib/spark/python/pyspark/streaming/kinesis.py in createStream(ssc, kinesisAppName, streamName, endpointUrl, regionName, initialPositionInStream, checkpointInterval, storageLevel, awsAccessKeyId, awsSecretKey, decoder)
     85             if 'ClassNotFoundException' in str(e.java_exception):
     86                 KinesisUtils._printErrorMsg(ssc.sparkContext)
---> 87             raise e
     88         stream = DStream(jstream, ssc, NoOpSerializer())
     89         return stream.map(lambda v: decoder(v))

Py4JJavaError: An error occurred while calling o35.loadClass.
: java.lang.ClassNotFoundException: org.apache.spark.streaming.kinesis.KinesisUtilsPythonHelper
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:745)

when I tried to add the jar (which I download from Maven) to spark-submit (spark-submit --jars

I get the following error:

"Must specify a primary resource (JAR or Python or R file)"

is there away to workaround this?

Thanks,

Upvotes: 0

Views: 371

Answers (1)

Alex Mattson
Alex Mattson

Reputation: 33

/usr/bin/spark-submit --jars /usr/lib/spark/extras/lib/spark-streaming-kinesis-asl.jar might work better as it links to the latest version AFAIK

Upvotes: 1

Related Questions