CODEWITHSUNDEEP
c#azure-data-factoryazure-databricks.net-spark
Nikhil Sharma
Nikhil Sharma

Reputation: 43

How to run .Net spark jobs on Databricks from Azure Data Factory?

In Azure data factory, you have a Databricks Acvitiy. This activity supports running python, jar and notebooks. And These notebooks may be written in scala, python, java, and R but not c#/.net.

Is there inherent or direct support where I can write my .NET spark code and run it on Databricks from Data Factory?

Can I use .NET spark in Azure Databricks to its full extent?

Upvotes: 3

Views: 958

Answers (1)

Ed Elliott
Ed Elliott

Reputation: 6856

You specify that you want to launch a JAR file. The .NET for Apache Spark project uses a JAR file to start a listener which the .NET code then connects to.

The JAR is microsoft-spark-2.4.x-0.12.1.jar (depends on the version of spark and the .NET NuGet version). The Class to run is org.apache.spark.deploy.dotnet.DotnetRunner which you need to pass the correct parameters to, so that it starts your .NET application.

Ed

Upvotes: 2

Related Questions