CODEWITHSUNDEEP
python-3.xpycharmdatabricksdatabricks-connect
Rohit Mishra
Rohit Mishra

Reputation: 571

Execute databricks magic command from PyCharm IDE

With databricks-connect we can successfully run codes written in Databricks or Databricks notebook from many IDE. Databricks has also created many magic commands to support their feature with regards to running multi-language support in each cell by adding commands like %sql or %md. One issue I am facing currently is when I try to execute Databricks notebooks in Pycharm is as follows:

Where as in IDE from FILE_TO_IMPORT import XYZ works.

Again everytime I download Databricks notebook it comments out the magic commands and that makes it impossible to be used anywhere outside Databricks environment.

It's really inefficient to convert all databricks magic command everytime I want to do any developement.

Is there any configuration I could set which automatically detects Databricks specific magic commands?

Any solution to this will be helpful. Thanks in Advance!!!

Upvotes: 6

Views: 1361

Answers (1)

Rohit Mishra
Rohit Mishra

Reputation: 571

Unfortunately, as per the databricks-connect version 6.2.0-

" We cannot use magic command outside the databricks environment directly. This will either require creating custom functions but again that will only work for Jupyter not PyCharm"

Again, since importing py files requires %run magic command so this also becomes a major issue. A solution to this is by converting the set of files to be imported as a python package and add it to the cluster via Databricks UI and then import and use it in PyCharm. But this is a very tedious process.

Upvotes: 4

Related Questions