CODEWITHSUNDEEP
scalasocketsapache-sparkspark-structured-streaming
nitinr708
nitinr708

Reputation: 1467

use readStream in Spark 1.6?

I am noob in spark, trying to read the data from a socket in Spark 1.6.0.

Following is my code doing is -

val sqc = new org.apache.spark.sql.SQLContext(sc)
val socketop = sqc.readStream.format("socket").option("myhost","localhost").option("port",1111).load

But I receive error as shown below -

readStream is not a member of org.apache.spark.sql.SQLContext

What am I doing wrong ?

Upvotes: 0

Views: 664

Answers (2)

Jacek Laskowski
Jacek Laskowski

Reputation: 74759

readStream.format("socket")

It won't work in Spark 1.6.0 as Spark Structured Streaming that this code is part of is only available as of Spark 2.0 and later.

Quoting Input Sources:

In Spark 2.0, there are a few built-in sources.

Socket source (for testing) - Reads UTF8 text data from a socket connection. The listening server socket is at the driver. Note that this should be used only for testing as this does not provide end-to-end fault-tolerance guarantees.

In order to read "the data from a socket in Spark 1.6.0" you have to use Spark Streaming (which I personally consider dead and recommend not using it unless you have to).

Upvotes: 1

Ambuj
Ambuj

Reputation: 92

Refer this it is using spark 1.6, it may help you https://spark.apache.org/docs/1.6.0/streaming-programming-guide.html#a-quick-example

Upvotes: 1

Related Questions