CODEWITHSUNDEEP
apache-sparkspark-streaming
neoguy
neoguy

Reputation: 61

Zero Input rate in Spark file streaming

I am using textFileStream for streaming data from Amazon s3. It gives me output only for the first batch. In addition to that, input rate in Spark UI is always zero. My question is similar to Spark Streaming: Input Rate and File stream [0] has "Avg: 0.00 events/sec" always

Any help would be appreciated.

Thanks.

Upvotes: 1

Views: 710

Answers (2)

harel
harel

Reputation: 525

I witness the same behavior. My guess is that the UI is basing these metrics on receivers and since textFileStream is not actually back by a receiver ( "file streams do not require running a receiver" ), the number is ( wrongly ) 0. This makes sense as the same behavior is seen when using the Kafka stream which also is not backed by a receiver.

Upvotes: 2

stevel
stevel

Reputation: 13440

EMR and their s3:// or Apache Spark and the s3n/s3a clients?

Upvotes: 0

Related Questions