CODEWITHSUNDEEP
pythonapache-sparkazure-databricksdbutils
play_something_good
play_something_good

Reputation: 141

use the output of notebook1 in notebook2

I have defined the class in notebook2 which should take the parameters from the notebook1. Because of that reason Im using the dbutils.notebook.run(path, 60, {"CATALOG": f"{CATALOG}"}) command to call notebook2. Note that with %run i was not able to pass the parameters using the widgets.

Now, I dont have problem with passing the parameters but the function I have defined in my class I want to use them in notebook1 but I cant. As I understand, dbutils.notebook.run start new spark session in notebook2 and thats the reason why I get the error that name 'gpg_encryption' is not defined.

Here is what Im doing:

1. running notebook1

dbutils.notebook.run(path, 60, {"CATALOG": f"{CATALOG}", "STORAGEACCOUNT_NAME": f"{STORAGEACCOUNT_NAME}"})

2.

Above code tiggering notebook2 and the class defined in there

  class gpg_encryption(gnupg.GPG):
  """
  Expands the GnuPG library:
  - Importing of BASE64 encoded keys.
  - Decrypting csv files to pandas dataframes.
  - Using simplified key-management.
  On databricks clusters.
  """
  
  asc_key_path = f'/Volumes/{CATALOG}/bronze/private'
  
  def __init__(self):
    self.gpg = gnupg.GPG()
    self.gpg.encoding = 'utf-8'
    self.create_key_path_dir()
    
  def create_key_path_dir(self):
    """
    Creates a folder to store keyfiles.
    """
    if not os.path.exists(self.asc_key_path):
      spark.sql(f"CREATE EXTERNAL VOLUME lakehouse_dev.bronze.private LOCATION 'abfss://bronze@{storage_account}.dfs.core.windows.net/private'")

3. After class is defined, in the same notebook (notebook2), running this command to get the output of the class in notebook1

dbutils.notebook.exit(gpg_encryption)

4. As a last step

running in notebook1

gpg_handler = gpg_encryption()

But I get the error

name 'gpg_encryption' is not defined

Upvotes: 0

Views: 36

Answers (1)

play_something_good
play_something_good

Reputation: 141

I just have expanded my init function as below

  def __init__(self, storage_account, catalog):
    self.gpg = gnupg.GPG()
    self.gpg.encoding = 'utf-8'
    self.storage_account = storage_account
    self.catalog = catalog
    self.asc_key_path = f'/Volumes/{self.catalog}/bronze/private'
    self.create_key_path_dir()

Then I run the notebook using %run, and after that I pass the parameters when calling the class itself gpg_handler = gpg_encryption(catalog = CATALOG, storage_account=STORAGEACCOUNT_NAME )

But would be nice if someone can explain, how it could be done in as explained in my question or in another way.

Upvotes: 0

Related Questions