CODEWITHSUNDEEP
pythonjsonazure-functionsazure-data-factoryazure-databricks
Nomad18
Nomad18

Reputation: 161

Can we pass Databricks output to function in an ADF Job?

Can anyone help me with Databricks and Azure function? I'm trying to pass data bricks JSON output to azure function body in ADF job, is it possible? If yes, How? If No, what other alternatives to do the same?

Upvotes: 8

Views: 16666

Answers (2)

Ed Elliott
Ed Elliott

Reputation: 6856

In the Notebook you can use:

dbutils.notebook.exit(myReturnValueGoesHere) (as already mentioned)

and then in ADF the JSON is an object that sits on output.runOutput, so @activity('RunNotebookActivityName').output.runOutput.

If you return:

dbutils.notebook.exit('{"hello": {"some": {"object": "value"}}}')

you can read in ADF using:

@activity('RunNotebookActivityName').output.runOutput.hello.some.object

Cool hey?

For a full list of what you can do see:

https://the.agilesql.club/2020/02/passing-status-messages-and-results-back-from-databricks-to-adf/

ed

Upvotes: 21

MartinJaffer-MSFT
MartinJaffer-MSFT

Reputation: 728

In Azure Databricks, there is a way to return a value on exit. dbutils.notebook.exit(myReturnValueGoesHere)

In Azure Data Factory V2, the DatabricksNotebook activity outputs JSON with 3 fields: "runPageUrl" , a URL to see the output of the run. "effectiveIntegrationRuntime" , where the code is executing "executionDuration"

If you use the above dbutils call, a fourth field will appear in the output JSON "runOutput" , where the "myReturnValueGoesHere" is displayed.

If you do not use the above dbutils call, you can scrape information from the runPageUrl.

Upvotes: 3

Related Questions