Saving files to Blob Storage

Question

I am running an API call using PowerShell via Azure Automation and I wanted to save the JSON files directly into a Blob Storage without having to save anything to the local machine first and then transfer the JSON files into Blob Storage as this process will be used by others. Is saving JSON files directly to Blob Storage possible without hitting the local machine? I am happy to change my approach to achieve to save JSON files directly into Blob Storage.

UPDATED API Code

$access_token ="Access_Token"
$URI =  "https://XXXXX"
$headers = @{“authorization” = “Bearer $access_token”} 
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$result = Invoke-RestMethod -Uri $URI -Headers $headers -ContentType $ContentType |ConvertTo-Json
$Result|ConvertFrom-Json| Select -ExpandProperty Forms

RESULT:

{
    "quiz": {
        "sport": {
            "q1": {
                "question": "Which one is correct team name in NBA?",
                "options": [
                    "New York Bulls",
                    "Los Angeles Kings",
                    "Golden State Warriros",
                    "Huston Rocket"
                ],
                "answer": "Huston Rocket"
            }
        },
        "maths": {
            "q1": {
                "question": "5 + 7 = ?",
                "options": [
                    "10",
                    "11",
                    "12",
                    "13"
                ],
                "answer": "12"
            },
            "q2": {
                "question": "12 - 8 = ?",
                "options": [
                    "1",
                    "2",
                    "3",
                    "4"
                ],
                "answer": "4"
            }
        }
    }
}

Answer 1

You can do this by using azure blob storage sdk in runbook.

First, you need to import the Microsoft.WindowsAzure.Storage.dll in azure blob storage sdk as module in the runbook. Follow the steps below:

1.Get the Microsoft.WindowsAzure.Storage.dll. If you don't know how to get it, just open the visual studio -> create a .net framework console project -> then right click the console project -> Manage Nuget Packages, then search and download the azure blob storage sdk WindowsAzure.Storage

2.When the package installed, build the console project, then in the bin -> debug folder, you can see Microsoft.WindowsAzure.Storage.dll.

3.Put the Microsoft.WindowsAzure.Storage.dll into a zip file named Microsoft.WindowsAzure.Storage.zip

4.Go to azure portal -> your automation account -> in the left pane, click "Modules" -> Add a module -> select the .zip file in step 2. You need wait for a few minutes for the module completes uploading(when it completes uploading, you can find the status of Microsoft.WindowsAzure.Storage module is Available ), see the screenshot below:

Secondly, create a powershell runbook, and write the code like below. Here, I just upload a string using UploadText("your_string_text") method. Note that since the sdk provides many upload method, you should check the return value from the api is text / stream / byte format, then select the proper upload method like UploadFromStream(your_stream) / UploadFromByteArray(byte[] buffer, int index, int count):

Write-Output "start the test"

Add-Type -Path "C:\Modules\User\Microsoft.WindowsAzure.Storage\Microsoft.WindowsAzure.Storage.dll"

$access_token ="Access_Token"
$URI =  "https://XXXXX"
$headers = @{“authorization” = “Bearer $access_token”} 
[Net.ServicePointManager]::SecurityProtocol = 
[Net.SecurityProtocolType]::Tls12
$result = Invoke-RestMethod -Uri $URI -Headers $headers -ContentType $ContentType |ConvertTo-Json
$blob_content_to_upload = $Result|ConvertFrom-Json| Select -ExpandProperty Forms

$account_name = "xxx"
$account_key = "xxx"
$container_name = "test1"
$blob_name = "testfile3.txt"

Write-Output "start communicate with blob storage"

$creds = New-Object "Microsoft.WindowsAzure.Storage.Auth.StorageCredentials" -ArgumentList $account_name,$account_key

$storageAccount = New-Object "Microsoft.WindowsAzure.Storage.CloudStorageAccount" -ArgumentList $creds,$true
$cloudBlobClient = $storageAccount.CreateCloudBlobClient()
$cloudBlobContainer = $cloudBlobClient.GetContainerReference($container_name)
$myblob = $cloudBlobContainer.GetBlockBlobReference($blob_name)

#note that the sdk also provides other methods like UploadFromStream(your_stream) / UploadFromByteArray(byte[] buffer, int index, int count), you can choose the proper method for your purpose.
$myblob.UploadText($blob_content_to_upload)

Write-Output("***the test is completed***")

Then you can run the runbook, after it completes, you can see the blob is created on blob storage with proper content. Note that during the running, there maybe a error says the modules loading issue, but it doesn't matter.

The test result, blob is created on blob storage.

If you want to specify the content-type, just add this line of code: $myblob.Properties.ContentType = "your_content_type"

Answer 2

I don't think we can export the json file directly to Blob Storage, we have to use the middle bridge (staging storage) to pass the file to Blob storage. Refer to Quickstart: Upload, download, and list blobs by using Azure PowerShell for details.

If you mean in Azure DevOps pipeline, then you can export the json file to a staging folder in the agent, then use the Azure File Copy task to upload the file to Blog storage.