Connect to Azure Storage Account from Azure Databricks job to fetch the data using MSI Authentication

Question

We have a job in Azure DataBricks which connects to a Azure Data Lake to fetch the data using below code. Now we have to convert it into MSI Managed Identity authentication instead of Service principal

we have to implement the authentication with out client id and secret

clientId = dbutils.secrets.get(scope = "plsadbsecret", key = "cltid")
clientSecret = dbutils.secrets.get(scope = "plsadbsecret", key = "cltsecret")
storageAccount = "xxxxxxxxxxxxxxxxxx"
folderPath = "xxxxxxxxxxxxxxxxxxxxxxx"
safeLimitFolderPath = "xxxxxxxxxxxxxxxxxxxxxxxxx"
spark.conf.set("fs.azure.account.auth.type." + storageAccount + ".dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type." + storageAccount +    ".dfs.core.windows.net",  "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id." + storageAccount + ".dfs.core.windows.net",   clientId)
spark.conf.set("fs.azure.account.oauth2.client.secret."+ storageAccount +  ".dfs.core.windows.net",  clientSecret)
spark.conf.set("fs.azure.account.oauth2.client.endpoint." + storageAccount +    ".dfs.core.windows.net", "https://login.microsoftonline.com/xxxxxxx-xxxx-xxxx-xxx- xxxxxxx/oauth2/token")   

 assetPath =xxxxxxxxxxxxxxx
 getEmployees= spark.read.format("delta").load(assetPath)



 Expecting to implement the MSI authentication