CODEWITHSUNDEEP
scalacsvapache-sparkspark-streaming
Levi
Levi

Reputation: 83

Spark Scala Streaming CSV

I am new in Spark/Scala. I know how to load CSV files:

    sqlContext.read.format("csv")

and how to read text streams and file streams:

    scc.textFileStream("""file:///c:\path\filename""");
    scc.fileStream[LongWritable, Text, TextInputFormat](...)

but how to read text stream in CSV format? Thanks, Levi

Upvotes: 4

Views: 5221

Answers (2)

Naman Agarwal
Naman Agarwal

Reputation: 654

You can stream your Csv file easily by using spark 2.2 structured streaming.

You can refer here

Upvotes: -1

Sudheer Palyam
Sudheer Palyam

Reputation: 2519

Here you go: 

val ssc = new StreamingContext(sparkConf, Seconds(5))


    // Create the FileInputDStream on the directory
    val lines = ssc.textFileStream("file:///C:/foo/bar")

    lines.foreachRDD(rdd => {
        if (!rdd.isEmpty()) {
          println("RDD row count: " + rdd.count())
         // Now you can convert this RDD to DataFrame/DataSet and perform business logic.  

        }
      }
    })

    ssc.start()
    ssc.awaitTermination()
  }

Upvotes: 6

Related Questions