CODEWITHSUNDEEP
scalaapache-sparkapache-kafkaspark-structured-streaming
Wilson Wii
Wilson Wii

Reputation: 101

Cannot resolve Queries with streaming sources must be executed with writeStream.start() Scala

I've trouble to resolve the following exception "Queries with streaming sources must be executed with writeStream.start();; kafka"

My code is the following :

val spark = SparkSession
      .builder()
      .getOrCreate()

val bootstrapServers = "localhost:9092"
val topicName = "name"

val df = spark
      .readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", bootstrapServers)
      .option("subscribe", topicName)
      .option("group.id", "Structured-Streaming-kpi")
      .option("failOnDataLoss", false)
      .load()

df.writeStream
      .format("console")
      .start()
      .awaitTermination();
df.show()

Upvotes: 5

Views: 4757

Answers (1)

soMuchToLearnAndShare
soMuchToLearnAndShare

Reputation: 1035

I think the problem is this df.show()

Your df is written to console already.

Try to remove it and see what happens

Upvotes: 3

Related Questions