U-SQL User Defined Function in Azure Data Factory

Question

I'm currently using Azure Data Factory for an ETL job and in the end I want to start a U-SQL job. I've created my datalake.usql script and the UDF's the script uses are in the datalake.usql.cs file, the same structure a U-SQL project has in Visual Studio (which is where I developed the U-SQL job and succesfully ran it).

After that, I uploaded them both to Azure Blob Storage and set up the U-SQL step in Azure Data Factory to use the U-SQL script, but it doesn't see the datalake.usql.cs with my UDF's.

How can I do this?

Answer 1

I figured this one out so I'll post the answer here if someone else has this problem in the future.

When you're developing locally, you have a script.usql and a script.usql.cs file. When you run it, Visual Studio does all the heavy lifting, compiles it all in a usable way and your script runs. But when you're trying to run a script from Azure Data Factory, you can't perform that compilation on the fly, like Visual Studio does.

The solution to this problem is to make an Assembly, a .dll file, from your script.usql.cs, upload it to the data storage you're using and register it in the U-SQL Catalog. Then, you can reference this assembly and use it normally, as you would on your local machine.

All the steps needed for this are presented in these short guides:

https://saveenr.gitbooks.io/usql-tutorial/content/usql-catalog/intro.html

https://saveenr.gitbooks.io/usql-tutorial/content/usql-catalog/usql-databases.html

https://saveenr.gitbooks.io/usql-tutorial/content/usql-catalog/assemblies.html

https://www.c-sharpcorner.com/UploadFile/1e050f/creating-and-using-dll-class-library-in-C-Sharp/