CODEWITHSUNDEEP
pythonapache-sparkpysparkazure-databricks
jozefow
jozefow

Reputation: 786

DataBricks adds # MAGIC in front of every python notebook line

My setup consist git repository connected to Azure DataBricks, this allows to develop databricks python notebooks in PyCharm, push to repo and run updated notebook in DataBricks UI.

Whenever I make change of notebook file from WebUI and do the commit, all lines in notebook file are prepended with # MAGIC:

# Databricks notebook source
%pip install -r "../requirements.txt"

import os

becomes:

# Databricks notebook source
# MAGIC %pip install -r "../requirements.txt"
# MAGIC 
# MAGIC import os

How to fix commits from Azure DataBricks to not change file on push?

Upvotes: 4

Views: 4594

Answers (1)

jozefow
jozefow

Reputation: 786

This is related to the way Azure DataBricks mixes magic commands and python code. When notebook (from Azure DataBricks UI) is split into separate parts, one containing only magic commands %sh pwd and others only python code, committed file is not messed up.

From text file, separate parts looks as follows:

# Databricks notebook source
# MAGIC %sh pwd
# COMMAND --------

<python code goes here>

Upvotes: 3

Related Questions