Using Command Line from C# to copy container using AzCopy

Question

I'm trying to copy containers in Azure from one storage location to another. I'm using the AzCopy command for this. First I get a list of all the containers and then run AzCopy based on the container name from the command line, using c# code.

The problem that I was running into is that it does copy the containers from one location to another but after 4 containers, it seems to get stuck. And the test keeps running forever. When I cancel the test I see all the other containers get copied as well.

I was wondering how can I solve this issue of having the test be complete and all the folders get copied over. I tried to wait after each call to make sure there is enough time for the call to complete. Also tried using cmd.WaitForExit(); after each call but that just gets stuck.

Any suggestions on what I could be missing, one thing I wanted to do was get the output after each call, because right now it only outputs the result once all the commands are finished. Also was thinking of how to run the command lines call sequentially so run only after the first has finished.

Any help would be appreciated!

namespace Test2
 {
[TestFixture]
class ContainerList
{
    [Test]
    public void CopyingContainerData()
    {
        CloudStorageAccount sourceCloudStorageAccount =
         CloudStorageAccount.Parse("StorageAccountKey");
         CloudBlobClient sourceCloudBlobClient = sourceCloudStorageAccount.CreateCloudBlobClient();
         List<string> outputLines = new List<string>();
         IEnumerable<CloudBlobContainer> containers = sourceCloudBlobClient.ListContainers();


         Process cmd = new Process();
         cmd.StartInfo.FileName = "cmd.exe";
         cmd.StartInfo.RedirectStandardInput = true;
         cmd.StartInfo.RedirectStandardOutput = true;
         cmd.StartInfo.CreateNoWindow = false;
         cmd.StartInfo.UseShellExecute = false;
         cmd.Start();
         int i = 0;

         foreach (CloudBlobContainer oneContainer in containers)
         {

                 string outputLine = oneContainer.Name;
                 outputLines.Add(outputLine);
                 string container = oneContainer.Name;
               string strCmdText =   @"AzCopy /Source:https://location1.blob.core.windows.net/" + container + @" /Dest:https://location2.blob.core.windows.net/" + container + @" /SourceKey:abc /DestKey:abc123 /S /NC:8 /XO /Y";
                 string location = @"cd C:\Program Files (x86)\Microsoft SDKs\Azure\AzCopy";

                 cmd.StandardInput.WriteLine(location);
                 cmd.StandardInput.WriteLine(strCmdText);
                 //System.Threading.Thread.Sleep(20000);
                 //cmd.WaitForExit();

                 i++;

             if (i == 15)
             {
                 break;
             }

         }
        string[] outputText = outputLines.ToArray();
        File.WriteAllLines(@"C:\AzureTests\CopyData.txt", outputText);


         cmd.StandardInput.Flush();
         cmd.StandardInput.Close();
         Console.WriteLine(cmd.StandardOutput.ReadToEnd());



    }
}
 }

Answer 1

We had similar scenario of AzCopy.exe invocation not returning control to .Net. The reason was parallel execution of AzCopy.exe without specifying the journal files. It share the journal files by default and gets access violation. When we issued different journal files to different instances of AzCopy it stared working.

Running multiple instances of AzCopy.exe from .Net

Answer 2

I suggest you use powershell to do this:

$SourceStorageAccount = "sourceStorageAccount"
$SourceStorageKey = "sourceKey"
$DestStorageAccount = "destStorageAccount"
$DestStorageKey = "destKey"

$SourceStorageContext = New-AzureStorageContext –StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey
$DestStorageContext = New-AzureStorageContext –StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey

$containers = Get-AzureStorageContainer -Context $SourceStorageContext

foreach($container in $containers) {  
    New-AzureStorageContainer -Context $DestStorageContext -Name $container.name -Permission Off

    $Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $container.name

    #Do the copy of everything
    foreach ($Blob in $Blobs) {
       Write-Output "Moving $Blob.Name"
       Start-CopyAzureStorageBlob -Context $SourceStorageContext -SrcContainer $container.name -SrcBlob $Blob.Name `
          -DestContext $DestStorageContext -DestContainer $container.name -DestBlob $Blob.Name
    } 
}