CODEWITHSUNDEEP
apache-sparkapache-beam
Sai Krishna
Sai Krishna

Reputation: 11

Apache beam word count example with spark runner fails with " Unknown 'runner' specified 'SparkRunner'"

I am trying to do spark-submit of the Apache beam word-count example by giving the below command

spark-submit --class org.apache.beam.examples.WordCount word-count-beam-0.1.jar --inputFile=pom.xml --output=counts --runner=SparkRunner

I get the below Exception:

Exception in thread "main" java.lang.IllegalArgumentException: Unknown 'runner' specified 'SparkRunner', supported pipeline runners [DirectRunner]

Upvotes: 1

Views: 960

Answers (2)

David Corley
David Corley

Reputation: 718

Looks like you're not building an Uber-jar with the necessary Spark dependencies. Re-run your Maven package as follows: mvn package -Pspark-runner This will build a Jar in target containing the wordcount classes as well as all of the necessary spark dependencies called something like: word-count-beam-bundled-0.1.jar Then use that jar in the spark-submit command

Upvotes: 0

Ben Chambers
Ben Chambers

Reputation: 6130

Your pom.xml needs to include a dependency on the Spark runner. The documentation on using the Spark runner includes more details about what is necessary.

Upvotes: 1

Related Questions