CODEWITHSUNDEEP
scalaapache-sparkdatabricks
Hagai
Hagai

Reputation: 275

Spark Create a dataframe from an InputStream?

I want to avoid writing the entire stream to a file and then load it to dataframe. what's the right way?

Upvotes: 4

Views: 3945

Answers (1)

Ramesh Maharjan
Ramesh Maharjan

Reputation: 41987

You can check Spark Streaming and sqlnetworkWordCount which explains that your problem can be solved by creating singleton instance of SparkSession by using SparkContext of SparkStreaming.

You should have better ideas by going through above links where dataframes are created from streaming rdd.

Upvotes: 1

Related Questions