CODEWITHSUNDEEP
apache-sparkkubernetesgoogle-cloud-platform
dex
dex

Reputation: 77

spark-submit to kubernetes cluster on Google cloud got 401 unauthorized

It's pretty standard spark-submit action, what's weird is that I got 401 from time to time, if I just wait a few minute, I can run again, until next time I got a 401.

it's very similar to this issue https://jira.apache.org/jira/browse/SPARK-24227

I'm just wondering if I'm not setting the GOOGLE_APPLICATION_CREDENTIAL right when submitting, but at the very beginning I haven't set any credential and can still run.

Anyone has similar issues?

Upvotes: 0

Views: 454

Answers (1)

Bob T.
Bob T.

Reputation: 43

It's a subtle bug with Spark couldnt refresh the access token. One work around is:

  • Run a dummy kubectl command to refresh access token. E.g: kubectl get namespace

Upvotes: 1

Related Questions