CODEWITHSUNDEEP
pythonazure-data-factorydatabricksazure-databricksspark-notebook
Simon Norton
Simon Norton

Reputation: 125

databricks Python notebook from azure data factory or locally if statement

I have a Databricks Python notebook that reads in a parameter from ADF using:

Program_Name = dbutils.widgets.get("Program_Name")

Is there an IF statement or something similar I can do in the notebook code, such that when I run the notebook interactively, it will substitute the call to dbutils with a plain assignment? Logically I want something like:

if running in ADF:
   Program_Name = dbutils.widgets.get("Program_Name")
else:
   Program_Name = 'ABC123'

If such a thing is possible, it beats the alternative of having to comment out the dbutils call every time I modify the rest of the notebook :) I've done similar things so that a script can be run from Jupyter/PyCharm or from the command line, but am not aware of anything that tells the python interpreter it's been called from ADF.

Many thanks!

Upvotes: 1

Views: 1109

Answers (2)

Alex Ott
Alex Ott

Reputation: 87279

Instead of catching up exception, it's just easier to create widget explicitly, and set default value (see docs).

dbutils.widgets.text("Program_Name", "ABC123", "Program name")
Program_Name = dbutils.widgets.get("Program_Name")

This has following benefits:

  1. The code is simpler - you don't need to have do any try/catch
  2. If necessary you can pass another program name even if you run notebook interactively

Upvotes: 2

Axel R.
Axel R.

Reputation: 1300

You could call the notebook with a prefix when running it from ADF. For example, you would call the notebook with Program_Name = "ADF_prog_name"

Program_Name = dbutils.widgets.get("Program_Name")
if not(Program_Name.contains('ADF')):
   Program_Name = 'ABC123'

Upvotes: 0

Related Questions