Get output from a Databricks notebook using Data Factory

Question

I have a Databricks notebook (using R) that It is used in a Data Factory pipeline to modify some data. Is there some way to obtain full output of the chunks (or from a specific chunk) in a file or directly in Data Factory, in case of errors?

Answer 1

When Databricks activity fails in azure data factory, it will not give the actual error message of the notebook instead gives the databricks execution failed with terminated.

To retrieve the notebook error message, connect the notebook activity upon failure to the set variable activity to store the error message using the expression: @activity('Notebook1').error.message