CODEWITHSUNDEEP
apache-sparkapache-beam
vipin
vipin

Reputation: 162

Apache beam StateSpec in spark

I am using StateSpec in beam and running that with spark runner

I am getting following error

Exception in thread "main" java.lang.UnsupportedOperationException: Found StateId annotations on com.walmart.rum.StreamingApp$3, but DoFn cannot yet be used with state in the SparkRunner. at org.apache.beam.runners.spark.translation.TranslationUtils.rejectStateAndTimers(TranslationUtils.java:269)

So does this means we StateSpec is not entirely supported or some version it is supported ?

Upvotes: 0

Views: 150

Answers (1)

Alexey Romanenko
Alexey Romanenko

Reputation: 1443

States and Timers are supported only in Batch mode and not supported in Streaming mode for Spark runner. It seems that you have streaming application and that is why it's failing.

Upvotes: 1

Related Questions