CODEWITHSUNDEEP
apache-sparkspark-streaming
kikulikov
kikulikov

Reputation: 2583

How to get a computed result from Spark?

I'm learning Spark and trying to build a simple service which should get computed results from Spark and return it to the user. There are a lot of good examples on Spark website on how to process streaming data. But I can't find any docs on how to retrieve the computed result from it.

For e.g. I have a Kafka queue and a Spark job like to count words. How to show the result to the end user? Ideally, I'd like to have a restful service which serves /words-count HTTP endpoint. But it's not clear on how to call Spark from that service.

Any reference to the doc or code example would be great. Thank you.

Upvotes: 0

Views: 478

Answers (1)

Joe C
Joe C

Reputation: 15674

In general, you would want to use the streaming API if you want to be constantly pushing results to users as they become available. There is a foreachRDD into which you can provide a function that takes an RDD and publishes its result. That function will be called every time the stream updates its result.

dstream.foreachRDD(rdd => publish(rdd))

Upvotes: 1

Related Questions