CODEWITHSUNDEEP
apache-sparkstreamingdstream
pauly
pauly

Reputation: 51

How to create a DStream from a List of string?

I have a list of string, but i cant find a way to change the list to a DStream of spark streaming. I tried this:

val tmpList = List("hi", "hello")    
val rdd = sqlContext.sparkContext.parallelize(Seq(tmpList))   
val rowRdd = rdd.map(v => Row(v: _*))

But the eclipse says sparkContext is not a member of sqlContext, so, How can i do this? Appreciate your help, Please.

Upvotes: 3

Views: 2493

Answers (1)

Sachin Janani
Sachin Janani

Reputation: 1319

DStream is the sequence of RDD and it is created when you have register a received to some streaming source like Kafka. For testing if you want to create DStream from list of RDD's you can do that as follows:

val rdd1 = sqlContext.sparkContext.parallelize(Seq(tmpList))
val rdd2 = sqlContext.sparkContext.parallelize(Seq(tmpList1))
ssc.queueStream[String](mutable.Queue(rdd1,rdd2))

Hope it answers your question.

Upvotes: 3

Related Questions