CODEWITHSUNDEEP
azureapache-sparkpysparkdatabricksazure-databricks
FRG96
FRG96

Reputation: 151

How to change the Spark user running jobs in Azure Databricks?

I am using Spark on Azure Databricks 5.5. I submit Spark jobs through the Databricks workspace UI via Jobs, Notebooks, and Spark-submit. The jobs are being successfully submitted, and Databricks new clusters are being spawned or existing ones are being utilized. But, the user that is running the job on the executor nodes is root by default. Is it possible to change the user that runs the jobs on Azure Databricks(which inherently doesn't allow SSH access)?

Usually, when I use spark-submit CLI on a cluster with Shell access; I change the user by using sudo: sudo -u exampleuser spark-submit.... In this example, the user 'exampleuser' is present on all nodes of the cluster. So, I wanted to know whether it is possible to change the user running the Spark jobs in Azure Databricks clusters.

Upvotes: 2

Views: 3108

Answers (2)

FRG96
FRG96

Reputation: 151

After discussion with Azure Databricks team, The correct way to change the user running the spark jobs is by setting an environment variable HADOOP_USER_NAME during cluster creation. This changes the effective user running the Spark Job from root to the user $HADOOP_USER_NAME

Upvotes: 1

CHEEKATLAPRADEEP
CHEEKATLAPRADEEP

Reputation: 12788

By default, all users can create and modify jobs unless an administrator enables jobs access control. With jobs access control, individual permissions determine a user’s abilities. This topic describes the individual permissions and how to enable and configure jobs access control.

  • Job permissions

There are five permission levels for jobs: No Permissions, Can View, Can Manage Run, Is Owner, and Can Manage. The Can Manage permission is reserved for administrators. The table lists the abilities for each permission.

enter image description here

  • Enable jobs access control

Go to the "Admin Console" => Select the "Access Control tab" => Click the "Enable" button next to "Cluster and Jos Access Control" => Click "confirm" to confirm the change.

enter image description here

  • Configure job permissions

Note: You must be an administrator or have "Is Owner" permission to configure job submissions.

First, select the job which you want to change the user => Click Advanced => Click the "Edit" link next to permissions. => In the pop-up dialog box, assign job permissions via the drop-down beside user's name.

enter image description here

Hope this helps.

If this answers your query, do click “Mark as Answer” and "Up-Vote" for the same. And, if you have any further query do let us know.

Upvotes: 0

Related Questions