CODEWITHSUNDEEP
apache-sparkspark-structured-streaming
Markus Wilhelm
Markus Wilhelm

Reputation: 171

Prevent progress output from Structured Streaming in Spark Shell

I'm currently trying Structured Streaming in the Scala Spark Shell. My problem with it is, that it writes continuously progress messages I can't hide. Something like that:

[Stage 5:==================================================>    (182 + 2) / 200]

I can write commands in the console and access the stream, but those messages always overwrite the lines I'm currently writing. Is there a way to disable the progress output in the shell?

As reference, that's how I define and start the stream in the shell:

val streaming = spark.readStream.schema(dataSchema).option("maxFilesPerTrigger", 1).json("/user/mwilhelm/data/activity-data/")
val activityCounts = streaming.groupBy("gt").count()
val activityQuery = activityCounts.writeStream.queryName("activity_counts").format("memory").outputMode("complete").start()

Upvotes: 1

Views: 256

Answers (1)

Ajay Srivastava
Ajay Srivastava

Reputation: 1181

Launching spark shell with following configuration will stop displaying progress indicator in console -

./bin/spark-shell --conf spark.ui.showConsoleProgress=false

Upvotes: 1

Related Questions