CODEWITHSUNDEEP
pythonpysparkpython-importdatabricksazure-databricks
justin.cherveny
justin.cherveny

Reputation: 33

How do I Import a class from library in databricks?

I'm using databricks in azure to do some machine learning work and I'm trying to import a class from a specific library, but it seems to work differently than I'm used to. (I normally write python code in jupyter notebook)

I am trying to run the following in a python notebook in databricks 

    from statsmodels.tsa.holtwinters import ExponentialSmoothing

I can import statsmodels by itself just fine, but trying to get this specific class gives me:

ImportError: No module named 'statsmodels.tsa.holtwinters' 

It works to import classes directly in jupyter, where I am used to writing python. Moreover, even though I have imported statsmodels, I can't directly call any of the classes in the actual code.

Is there something special I have to do in databricks to use a specific class from a library?

Upvotes: 1

Views: 4559

Answers (2)

Skippy le Grand Gourou
Skippy le Grand Gourou

Reputation: 7734

It seems barely documented, but note that one can load a notebook as a library.

Having your library notebook named "mylib" as

def foobar():
    print("woohoo")

if you execute it in another notebook with (assuming it’s in the same directory)

%run ./mylib

you can access to its functions :

foobar()

Upvotes: 1

Peter Pan
Peter Pan

Reputation: 24148

At first, I tried to install statsmodels successfully via the left toolbar Clusters -> Interactive Clusters -> Libraries -> Install New -> PyPI, but got the same issue as yours.

Then I restarted my cluster and created a notebook to install it via %sh, as the figure below.

enter image description here

%sh
/databricks/python/bin/pip install -U statsmodels

It works without any issue.

Note: It's a temporary solution for installing. If you restart the cluster, the statsmodels installed module will disappear. So you have to run the command first to install statsmodels after restart the attached cluster.

Upvotes: 0

Related Questions