CODEWITHSUNDEEP
apache-sparkspark-streaming
Tao Li
Tao Li

Reputation: 303

Spark Streaming - How to get the "Key" in updateStateByKey function

I'm using the updateStateByKey function in Spark Streaming application to persist and update the status for each key. The question is I want to know the "key" inside the update function.

input.updateStateByKey(updateStateByKeyOfUsers)

def updateStateByKeyOfUsers(newUsers: Seq[Set[String]],
                          userStatus: Option[(#####)]
                           ): Option[(#####)] = {
   //How to get the "Key"
}

-Tao

Upvotes: 4

Views: 701

Answers (1)

Andrii Rubtsov
Andrii Rubtsov

Reputation: 523

Generally speaking, Spark API does not let you obtain a key. Which is quite sad. You have two options: either include a key in each of inputs or include it in a state.

Upvotes: 4

Related Questions