CODEWITHSUNDEEP
azureazure-devopsazure-pipelinesazure-synapse
R. Maier
R. Maier

Reputation: 428

Azure Synapse - Import python scripts in notebooks

Am using Azure Synapse in combination with jupyter notebooks: enter image description here

Many of my jupyter notebooks import some custom python scripts like the util-import: enter image description here

However, there's no option hold *.py files in Azure Synapse. Always when i use the import functionalty, the *.py is transformed to a notebook (On laptop it was util.py, after Synapse import it's a notebook):

enter image description here

How can custom *.py files be used in Azure Synapse Notebooks, without transforming them from *.py to notebook?

Upvotes: 0

Views: 2744

Answers (2)

zthatch56
zthatch56

Reputation: 43

I have found that the workspace imports generally require full packages, which doesn't seem to be what this question is asking for.

You can effectively import single files in R or python with .R and .py files respectively by reading the file from the associated abfss and then evaluating it with the corresponding python or R functions that can evaluate strings as code.

In R or python you can run

file_string = mssparkutils.fs.head("abfss://<container_name>@<storage_account>.dfs.core.windows.net/Path/to/file.<R or py>")

This will create a string variable containing the entire contents of the file. Then, in python, you can run

exec(file_string)

Or in R you would run

eval(parse(text = file_string))

Upvotes: 1

Joel Cochran
Joel Cochran

Reputation: 7728

There are several options for adding python to Synapse. You can manage them at the workspace, pool, or session level. The method I've had the most success with is loading from PyPi to a Spark Pool, but you can also upload Wheel or JAR files to your workspace and reference those in your notebooks. One caveat on using PyPi is that if the package has a dependency on C library packages, Synapse won't load it into your pool.

Upvotes: 1

Related Questions