CODEWITHSUNDEEP
azurecmdunzipazure-data-factoryazure-batch
Cargowire
Cargowire

Reputation: 1462

How do I Unzip and Execute a Batch Service job as part of Azure Data Factory

Azure Data Factory can execute custom activities as Batch Service Jobs. These jobs can run from an .exe (and associated dependencies) in a storage account which are copied across prior to execution.

There is a limitation on the files in the storage account that can be used:

Total size of resourceFiles cannot be more than 32768 characters

The solution appears to be to zip the files in the storage account and unzip as part of the command. This post suggests running the Batch Service Command in Azure Data Factory as:

Unzip.exe [myZipFilename] && MyExeName.exe [cmdLineArgs]

Running this locally on a Windows 10 machine works fine. Setting this as the Command parameter on the batch service custom activity (using a Cloud Services Windows Server 2019 OS Image App Pool) results in:

caution: filename not matched: &&

It feels like something basic that I'm missing but I've tried various permutations and cannot get it to work.

Upvotes: 1

Views: 2232

Answers (4)

Olga
Olga

Reputation: 49

Another option, which doesn't require copying additional files along with zip file:

cmd /c "tar -xf MyExe.zip & MyExe.exe"

tar -xf MyExe.zip & MyExe.exe gives an error, but cmd /c "tar -xf MyExe.zip & MyExe.exe" works fine on windows 10

Upvotes: 0

Daniel W.
Daniel W.

Reputation: 1198

I had the same problem and solved it with a short powershell script, that extracts my zip and start my Program. The generic script look like (called unzipandrun.ps1):

[CmdletBinding()]
param (    
    [Parameter(Mandatory = $true)][String]$zipFile,    
    [Parameter(Mandatory = $true)][String]$executable,    
    [Parameter(Mandatory = $true)][String]$prm
)
#######
# Unzip
#######
Add-Type -AssemblyName System.IO.Compression.FileSystem
function Unzip
{
    param([string]$zipfile, [string]$outpath)
    [System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
}
"Start Unzip $zipFile" | Write-Host
Unzip $zipFile $pwd
###########
# Start exe
###########
"Start $executable $prm" | Write-Host
$cmd = "& $executable $prm"
Invoke-Expression $cmd

So if I want to start myapp.exe with parameter /t 123 on azure batch I would start:

PowerShell -NoProfile -ExecutionPolicy Bypass -Command "unzipandrun.ps1 '/t 123'"

Upvotes: 0

Cargowire
Cargowire

Reputation: 1462

Without full knowledge of the context in which ADF runs Custom Activity Commands on a Windows Batch Service Node I changed my setup to avoid expecting Unzip.exe to exist (which it appears not to when running under cmd /c "Unzip.exe" rather than with just Unzip.exe as the command).

Now my storage account contents backing the custom activity has:

  • executable.zip (my .NET Core Console application published for windows with all dependencies)
  • unzip.exe (taken from Git Bash on my local machine)
    • including the msys-2.0.dll and msys-bz2-1.dll dependencies

The command in ADF is then:

cmd /c "Unzip.exe executable-with-deps.zip && executable.exe"

Upvotes: 2

Tats_innit
Tats_innit

Reputation: 34107

Sharing few possibilities which could be happenening, I agree that something small is missing, if you want to share the exact commandline you are trying then feel free to share.

Few ideas which could be causing this behavioour:

  • Please try your command-line under quotes from batch point of view like: cmd /c "Unzip.exe [myZipFilename] && MyExeName.exe [cmdLineArgs]"

  • Make sure your file exist i.e. one possibility is : that in the end the command is trying to run an empty string with && i.e. cmd /c "unzip.exe "empty" && ...

Hope one of the above 2 fixes, or feel free to add more detail et. al.

Upvotes: 3

Related Questions