CODEWITHSUNDEEP
pysparkpytestazure-databricksdbutils
Alexis Kan
Alexis Kan

Reputation: 81

Github Actions: Problem with dbutils when running pytest

I am trying to run a pytest using a function from my main code with Github Actions. My main code as a dbutils.widgets that cause my pytest crashing.

On my first iteration I got the error:

`NameError: name 'dbutils' is not defined`

So in my pytest file I added following lines of code and also pip installed pyspark in my Github Acrtions workflow before running pystest:

from pyspark.dbutils import DBUtils
dbutils = DBUtils(spark_session)

But I still got an error. From Github Actions log:

    from pyspark.dbutils import DBUtils
E   ModuleNotFoundError: No module named 'pyspark.dbutils`

This is strange because it successfully installed pyspark in the previous step.

How can I solve this dbutil issue? I don't even need it for my pytest.

Upvotes: 1

Views: 527

Answers (1)

Alex Ott
Alex Ott

Reputation: 87069

The dbutils isn't a part of PySpark - it's a Databricks-specific functionality and is available when a code is running on Databricks. This answer shows how you can work around this issue.

Upvotes: 1

Related Questions