CODEWITHSUNDEEP
yamlazure-pipelinesdatabricksazure-databricks
RVandakar
RVandakar

Reputation: 81

Azure DevOps pipeline integration with Databricks + how to print Databricks notebook result on pipeline result screen

Am trying to integrate Azure Databricks within DevOps pipeline and used following URL: https://menziess.github.io/howto/run/databricks-notebooks-from-devops/

  1. At Azure Databricks, created a job on top of my notebook
  2. Used a separate VM with preinstalled Databricks-CLI and set it as a self-hosted pool agent
  3. Prepared a YAML script in pipeline, which calls the above databricks job on my self hosted agent pool

My Pipeline and databricks jobs are executing successfully.

But I want to extract the output generated by notebook inside pepeline and print on console. In this way anyone can trigger the pipeline by passing their parameters and view the result on console screen. Notebook mostly returns output as table data (e.g. select * from table).

Could someone help me with this.

Upvotes: 0

Views: 996

Answers (2)

Menzies
Menzies

Reputation: 122

You could use the cli command databricks fs cp in a final step in the DevOps pipeline to copy the data to the agent, then print it out using an appropriate tool.

Upvotes: 0

Felix
Felix

Reputation: 1162

we can try to use the powerShell script to set the result as the variable, then we can output the data to a file. This can be a work around. At present, in the pipeline we can not pass their parameters and view the result on console screen.

Upvotes: 0

Related Questions