CODEWITHSUNDEEP
c#azureazure-blob-storageazure-batch
NDJ
NDJ

Reputation: 5194

moving files from a batch job to blob

I am executing an azure-batch job, which creates a zip file as its output. The batch is being executed by an orchestrator with the responsibility of moving the output files to blob. I have this working, but it feels clunky - i.e. I download the file locally to the orchestrator then upload to blob:

CloudTask task; // executed task...
var node = task.GetNodeFile(fileName);
using (var stream = File.OpenWrite(localFile))
{
  node.CopyToStream(stream);
}

var blobRef = _blobContainer.GetBlockBlobReference(blobFileName);
blobRef.UploadFromFile(localFile, FileMode.Open);

I tried passing the blob stream to the CopyToStream method directly, but nothing was moved to the blob:

node.CopyToStream(blobRef.OpenWrite());

Is it possible to copy the output file from a batch vm to blob without this extra hop?

Upvotes: 0

Views: 629

Answers (2)

NDJ
NDJ

Reputation: 5194

the answer was actually quite simple - disposing the cloud ref stream seems to force a flush - i.e.:

CloudTask task; // executed task...
var node = task.GetNodeFile(fileName);

using(var stream = _blobContainer.GetBlockBlobReference(blobFileName))
{
  node.CopyToStream(stream);
}

Upvotes: 0

Peter
Peter

Reputation: 27944

You can do this by using a memory stream:

CloudTask task; // executed task...
var node = task.GetNodeFile(fileName);

using (var ms = new MemoryStream())
{
    node.CopyToStream(ms);

    var blobRef = _blobContainer.GetBlockBlobReference(blobFileName);
    blobRef.UploadFromStream(ms);
}

Upvotes: 1

Related Questions