CODEWITHSUNDEEP
azure-powershellazure-automation
Steven
Steven

Reputation: 23

Azure Automation Power Shell- How to download Zip folder from URL and post to Azure blob

I am trying to download zip data from a url (https://data.police.uk/data/archive/2019-10.zip) and place into a Storage Account (blob).

The PS need to run within an Azure automation account.

When using the below, I get an exception

There is not enough space on the disk

$url = "https://data.police.uk/data/archive/2019-10.zip"
$output = "\policedata.zip"

$wc = New-Object System.Net.WebClient
$download = $wc.DownloadFile($url, $output)

Is there a way to move the zip file to an Azure storage blob within downloading it locally first, or extracting the CSV's out of it?

Upvotes: 2

Views: 480

Answers (1)

Kunal Deo
Kunal Deo

Reputation: 2308

Azure Automation is not the right solution here given the file you are downloading is 1.6 GB. The problem is the limits Azure Automation has:

Maximum amount of disk space allowed per sandbox: 1 GB

Maximum amount of memory given to a sandbox: 400 MB

You can neither download the file before uploading nor hold it in memory and then pipe it to the storage account.

You could try to setup a two way streaming where one stream downloads file and another stream uploads the file. But this will be an overkill for what you are trying to do. I would suggest using vanilla container or VM to automate this.

Upvotes: 0

Related Questions