How does partitioning of DStreams ( for updateStateByKey() ) work and how to verify it?

Question

I am using updateStateByKey() operation to maintain state in my Spark Streaming application. The input data is coming through a Kafka topic.

1. I want to understand how are DStreams partitioned?
2. How does the partitioning work with mapWithState() or updateStatebyKey() method?
3. In updateStateByKey() does the old state and the new values against a given key processed on same node ?
4. How frequent is the shuffle for updateStateByKey() method ?

The state I have to maintaining contains ~ 100000 keys and I want to avoid shuffle every time I update the state , any tips to do it ?

Answer 1

Link to the answer to the same question by Tathagat Das :

https://www.mail-archive.com/user@spark.apache.org/msg43512.html

Following is the text :

Both mapWithState() and updateStateByKey() by default uses the HashPartitioner, and hashes the key in the key-value DStream on which the state operation is applied. The new data and state is partition in the exact same partitioner, so that same keys from the new data (from the input DStream) get shuffled and colocated with the already partitioned state RDDs. So the new data is brought to the corresponding old state in the same machine and then the state mapping /updating function is applied.

The state is not shuffled every time, only the batches of new data is shuffled in every batch

Answer 2

updateStateByKey takes a Partitioner as it's second argument. http://spark.apache.org/docs/1.6.0/api/scala/index.html#org.apache.spark.streaming.dstream.PairDStreamFunctions