CODEWITHSUNDEEP
multithreadingapache-sparkpyspark
Manoj kumar
Manoj kumar

Reputation: 267

How to update the value of a shared variable in all exectors - PySpark

I'm using a Pyspark script to call an external authorized API. The authentication token of the script will expire in 2hrs. Since the script runs for more than expiry time, I'm trying to update the token when it is expired. Is it possible to update the access token variable across all executors?

Upvotes: 0

Views: 126

Answers (1)

Shane
Shane

Reputation: 626

You can use broadcast variables may be. Lets say if your token is valid for 3600 seconds (1 hour), you can update your broadcast variable at every 50th minute.

if (broadcastedToken != null) {
     broadcastedToken.destroy() // for every 3600 seconds
  }
      broadcastedToken = sparkSession.sparkContext.broadcast((apiToken))
}

In executors you can get the broadcasted token,

val apiToken = broadcastedToken.value

Upvotes: 1

Related Questions