CODEWITHSUNDEEP
apache-spark
Subba Rao
Subba Rao

Reputation: 165

How to update a large broadcast variable in a streaming use case?

I have a use case where I have a streaming job running getting input data from kafka queue. And I have a reference data of 1 million rows which gets updated every hour. I load the reference data in the driver and then broadcast it to the workers. I would like to update this broadcast variable (in the driver) and resend it to workers.

What would be the best way to do this within spark, without introducing hbase/redis/cassandra etc?

And how reliable is this?

Do let me know if more information is needed. Thank you in advance. =)

Upvotes: 5

Views: 1426

Answers (1)

Timofey Chernousov
Timofey Chernousov

Reputation: 1294

Answer to the similar question was given later here: How can I update a broadcast variable in spark streaming?

In short, you will need to: "unpersist" broadcast variable, update, and rebroadcast it.

PS. formally this question is not a duplicate, because it was posted earlier.

Upvotes: 3

Related Questions