CODEWITHSUNDEEP
apache-sparkpysparkspark-structured-streaming
SaSJo
SaSJo

Reputation: 81

Why Spark Structured Streaming is ideal for real-time operations?

I wanna construct a real-time application but I don't know if I should use Spark Streaming or Spark Structured Streaming.

I read online that Structured Streaming is ideal for real-time applications but is not clear why...

Can someone explain it?

Upvotes: 0

Views: 82

Answers (1)

QuickSilver
QuickSilver

Reputation: 4045

Spark Streaming works on something we call a micro batch. ... Each batch represents an RDD. Structured Streaming works on the same architecture of polling the data after some duration, based on your trigger interval, but it has some distinction from the Spark Streaming which makes it more inclined towards real streaming.

For developers all they need to worry is that Spark streaming you will you RDDs but in Spark Structured Streaming you get Dataframes and DataSet.
If you want so very low level(i.e. per record) operations go for RDDs(i.e. Spark Streaming) and but your application can build on Dataframes and querying them like SQL in real time then go for DataFrames(i.e. Spark Structured Streaming)

Eventually RDDs can be converted to Dataframes and vice versa

Upvotes: 1

Related Questions