CODEWITHSUNDEEP
azureazure-devopsdatabricksazure-databricks
romanzdk
romanzdk

Reputation: 1332

Databricks - push file to GIT

Is there any way to:

  1. Get file (.csv for example) from storage (ADLS)
  2. Push it to GIT (Azure DevOps) repository

using (Azure) Databricks notebook (programatically)?

I tried Databricks Repos, however it works only for notebooks and only via UI and git clone did not work in a notebook.

Upvotes: 2

Views: 1501

Answers (2)

Larsq
Larsq

Reputation: 365

I know databricks repos is quite limited in what git commands you can do (which should be improved in my humble opinion).

Anyway, by default, databricks only 'recognizes' notebook files in your repos. You can allow repos to detect other files as well. Under Settings -> Admin console enable the option Files in Repos. I think you'd need Admin Rights to be allowed to change this setting.

More info: https://databricks.com/blog/2021/10/07/databricks-repos-is-now-generally-available.html

Upvotes: 1

romanzdk
romanzdk

Reputation: 1332

What I did is:

  1. Copy file from desired location (ADLS) to some temp folder (e.g. FileStore) (using shell or dbutils)
  2. git clone in notebook using shell. cloning using personal access token from Azure DevOps. Files get cloned to driver node so they are not visible nor accessible anywhere (in UI).
  3. Copied wanted files to the cloned repo folder
  4. Pushed back to git (you have to set up user name and email using git config in advance)

Upvotes: 0

Related Questions