CODEWITHSUNDEEP
azure-data-lakeu-sql
aaronsteers
aaronsteers

Reputation: 2571

Run U-SQL Script without recompiling each time?

Is there a way to rerun a U-SQL job without ADLA having to recompile the job each time? I have many scripts which take 30-90 seconds to compile, and it would be awesome if there was a way to trim that compile time off of the total runtime.

Upvotes: 1

Views: 65

Answers (1)

Michael Rys
Michael Rys

Reputation: 6684

Unfortunately, this is not available yet.

Note that the job graph and optimization is highly dependent on the data distribution, data sizes, number of files etc, that are determined during the preparation phase and can change quite dramatically between different invocations.

Additionally, unlike a SMP database, the language is not interpreted but is compiled into stage level programs that get then deployed and run as part of the scale-out execution. So a lot of the time you see in the "compilation" phase is the actual code generation as well.

Having said that, we are working on improving the latency that is introduced by the preparation phase steps, including faster codegen, looking at job history etc. One recent big improvement in this area that recently has been released is the fast file set support for example.

Upvotes: 2

Related Questions