CODEWITHSUNDEEP
databricksazure-databricks
Patterson
Patterson

Reputation: 2821

How to Authenitcate Access to ADLS from Databricks without creating Service Principle with Using AD App Registrations to Mount a Drive

Is it possible to mount ADLS Gen2 account/container without configuring service principle credentials to access storage?

The following article states that it's possible to access a ADLS storage account with first configuring a service principle.

https://learn.microsoft.com/en-us/azure/databricks/data-governance/credential-passthrough/adls-passthrough

I normally mount a drive in Databricks by first creating a service principle by Registering an Azure AD application using App Registrations. I would use the following code to mount:

if check(mount)==1:
  resultMsg = "<div>%s is already mounted. </div>" % mount
else:
  dbutils.fs.mount(
  source = f"abfss://root@{Lake}.dfs.core.windows.net/",
  mount_point = mount,
  extra_configs = configs)
  resultMsg = "<div>%s was mounted. </div>" % mount

At present I am waiting for the administrator at my place of work to give me the permissions to register applications.

So, I was wondering if I could mount a drive from Databricks without first registering an application?

I tried @JayashankarGS suggestion as follows:

mount = "/mnt/lake"

if check(mount)==1:
  resultMsg = "<div>%s is already mounted. </div>" % mount
else:
  dbutils.fs.mount(
  source = f"abfss://root@{Lake}.dfs.core.windows.net/",
  mount_point = mount,
  extra_configs = configs)
  resultMsg = "<div>%s was mounted. </div>" % mount

But I got the error;

IllegalArgumentException: Unsupported Azure Scheme: abfss

Upvotes: 0

Views: 798

Answers (2)

JayashankarGS
JayashankarGS

Reputation: 8140

You use storage account key to access data in adls2. Below is code for it.

    spark.conf.set("fs.azure.account.key.<storage account name>.dfs.core.windows.net","Your stoarge account key")
    dbutils.fs.ls("abfss://<container name>@<storage account name>.dfs.core.windows.net/<dir>")

Output:

enter image description here

Or you can follow below approach.

    dbutils.fs.mount(source = "wasbs://[email protected]",
            mount_point = "/mnt/iotdata",
            extra_configs = {"fs.azure.account.key.jgsadls.blob.core.windows.net":"Your account key"})
    dbutils.fs.ls("/mnt/iotdata/json_data/")

This is the result.

enter image description here

Upvotes: 0

Bhagyashree
Bhagyashree

Reputation: 38

You can try to mount storage using account key

I did similar thing in my project as well!!

You can use below code as a reference

    Container_name = "root"
    storage_account = "lake"
    key = ""

    url = "wasbs://" + container_name + "@" + storage_account + ".blob.core.windows.net/"
    config = "fs.azure.account.key." + storage_account + ".blob.core.windows.net"

    mount_folder = "/mnt/lake"
    mounted_list = dbutils.fs.mounts()

    mounted_exist = False
    for item in mounted_list:
        if mount_folder in item[0]:
            mounted_exist = True
            break

    if not mounted_exist:
        dbutils.fs.mount(source = url, mount_point = mount_folder, extra_configs = {config : key})

Upvotes: 1

Related Questions