Uploading all Sub-Directories and Files to an Azure Storage Account File Share Using VB.NET

Question

How can I upload all files and sub-directories within a root directory on my PC, to my Azure Storage Account File Share (files, not blobs)? The idea being that I want to effectively "clone" all files and directory structure that exist on my PC or on a hard drive, and upload them onto my Azure File Share.

Dominic Whitham's Online Graphic Portfolio

Answer 1

I needed to upload thousands of files and their folders to my Azure Storage Account File Share (files, not blobs). So I created a simple PC app that would let me select a folder on my PC (or attached hard drive), click a button, and voila! The files and directories were then easily "cloned" to my Azure File Share location in the cloud.

To get started, you'll need to create a new project in Visual Studio. I'm using VS 2017.

Step 1: Create a new Windows Forms App

Step 2: We need to add a NuGet package, called WindowsAzure.Storage (v. 9.3.3), so go to Project-->Manage NuGet Packages. Click the Browse tab and search for WindowsAzure.Storage. Then click "Install".

Step 3: Add a new Class to your Project. Then copy/paste the following code, which contains the functions needed to do the heavy lifting of uploading your files...

Step 3a: Important; Be sure to replace the "yourStorageAccountName" and "yourStorageAccountKey" values with valid ones representing your Azure File Storage account.

Imports Microsoft.WindowsAzure.Storage
Imports System.IO
Imports Microsoft.WindowsAzure.Storage.File

Public Class AzureStorageUpload
    Private mConnectionString As String = "DefaultEndpointsProtocol=https;AccountName=yourStorageAccountName;AccountKey=yourStorageAccountKey"

    Private mUploadFileSize As Long
    Private mUploadFileExtension As String
    Private mCloudFile As CloudFile

    Public Sub UploadFileAsFileToAzureStorage(fileShare As String, folderPath As String, fullFilePath As String, originalName As String)
        'Connect to Azure
        Dim storageAccount As CloudStorageAccount = CloudStorageAccount.Parse(mConnectionString)

        ' Create a reference to the file client.
        Dim CloudFileClient = storageAccount.CreateCloudFileClient()
        Dim share As CloudFileShare = CloudFileClient.GetShareReference(fileShare)
        If share.Exists = True Then
            Dim rootDir As CloudFileDirectory = share.GetRootDirectoryReference()
            Dim cfd As CloudFileDirectory = share.GetRootDirectoryReference()
            cfd = cfd.GetDirectoryReference(folderPath)

            'Create a reference to the filename that you will be uploading
            mCloudFile = cfd.GetFileReference(originalName)

            'Upload the file to Azure
            mCloudFile.UploadFromFile(fullFilePath)

            mUploadFileSize = mCloudFile.Properties.Length
            mUploadFileExtension = Path.GetExtension(mCloudFile.Name)
        End If

        share = Nothing
        CloudFileClient = Nothing
        storageAccount = Nothing
    End Sub

    Public Sub CreateDirectory(fileShare As String, folder As String)
        'Connect to Azure
        Dim storageAccount As CloudStorageAccount = CloudStorageAccount.Parse(mConnectionString)

        ' Create a reference to the file client.
        Dim CloudFileClient = storageAccount.CreateCloudFileClient()
        Dim share As CloudFileShare = CloudFileClient.GetShareReference(fileShare)

        If share.Exists = True Then
            Dim cfd As CloudFileDirectory = share.GetRootDirectoryReference()
            cfd = cfd.GetDirectoryReference(folder)
            cfd.CreateIfNotExists()

            cfd = Nothing
        End If

        share = Nothing
        CloudFileClient = Nothing
        storageAccount = Nothing
    End Sub

    Private Function PathOnly(fileName As String) As String
        Dim l As Long

        l = InStrRev(fileName, "\")
        If l = 0 Then
            Return fileName
        Else
            Return Mid(fileName, 1, l)
        End If
    End Function

    Public Function JobOnly(ByVal MyJob As String) As String
        Dim l As Long

        JobOnly = MyJob
        l = InStrRev(JobOnly, "\")
        If l = 0 Then
            Exit Function
        End If
        JobOnly = Mid(JobOnly, l + 1, Len(JobOnly) - l)

        l = InStrRev(JobOnly, ".")
        If l = 0 Then
            Exit Function
        Else
            'Eliminate the extension
            JobOnly = Mid(JobOnly, 1, l - 1)
        End If
    End Function
End Class

Step 4: On your Windows Form (form1), place the following controls:

1. TextBox1 should be a wide textbox which will hold the desired path. This path will be the "root" path for all the files and
   sub-directories that will be created and uploaded to your Azure File Share. Note that the root folder's files (if you have any), will NOT be enumerated, and thus will not be copied, but all of it's
   subdirectories, their subdirectories, and files will all be "cloned", uploaded, and copied.
2. Place a button, Button1 that says "Select Folder"
3. Below that, put Button2 that says "Upload to Cloud"
4. Below that, put Button3 that says "Cancel"
5. Add a StatusStrip, docked at the bottom, and in it place a ToolStripStatusLabel whose name is "tsData1"
6. Add a FolderBrowserDialog control, and name it "fbd1"

Step 5: Now you're ready to add the code behind the form:

Imports System.IO

Public Class Form1
    Private mWorking As Boolean

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        ' Select Folder/Directory to Upload
        ' Note:  The idea is that all SubDirectories and Files will be copied (cloned) to your Azure Storage FileShare (as Files, not Blobs)
        Dim DiaResult As DialogResult
        Dim batchFolder As String = "c:\"

        With fbd1
            .RootFolder = Environment.SpecialFolder.MyComputer
            .SelectedPath = batchFolder
            .Description = "Select folder to Upload to Azure Storage"
            DiaResult = .ShowDialog()
            If DiaResult <> Windows.Forms.DialogResult.OK Then
                Exit Sub
            Else
                TextBox1.Text = .SelectedPath
            End If
        End With
    End Sub

    Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
        Dim i As Integer

        WaitCursor(True)
        ' Note: Here, I'm searching for all files that have a .png extension.  If you simply want to transfer ALL files within the folders,
        ' just use *.* instead of *.png
        ' Also note that the initial directory's files are NOT enumerated (so if there are FILES in your initial chosen directory, they are NOT copied)
        Dim files = From d In Directory.EnumerateDirectories(TextBox1.Text, "*", SearchOption.AllDirectories)
                    From f In Directory.EnumerateFiles(d, "*.*")
                    Select f

        Dim txtFilesArray() As String = files.ToArray()
        WaitCursor(False)

        Dim count As Integer = 0

        Dim asu As New AzureStorageUpload

        ' Get the initial directory which has already been selected
        Dim Prefix As String = LCase(TextBox1.Text)
        Prefix = Replace(Prefix, Path.GetDirectoryName(Prefix), "")

        ' Create a log file to record the progress...
        Dim myLog As String = GetAppPath() & "\UploadLog.txt"

        ' Your file share goes here.  Be sure everything is lower case
        ' Note that your file share must already exist (use the Azure portal to create your fileshare name)
        Dim yourFileShare As String = "yourfileshare"

        Try
            ' This will create your initial directory within your file share
            ' Your intention is that you'll be uploading (cloning) everything, including subdirectories and files,
            ' that currently exist on your PC inside your "starting" folder.
            asu.CreateDirectory(yourFileShare, Prefix)
        Catch ex As Exception

        End Try

        MsgBox("Ready to begin uploading files...")

        Dim sw2 As New StreamWriter(New FileStream(myLog, FileMode.Create, FileAccess.Write))
        Dim startTime As Date = DateAndTime.Now
        sw2.WriteLine("Starting Upload at " & startTime.ToLongTimeString & " on " & startTime.ToLongDateString)

        mWorking = True

        WaitCursor(True)
        While mWorking = True
            For i = 0 To txtFilesArray.Length - 1
                Application.DoEvents()

                Dim origPath As String = txtFilesArray(i)

                If File.Exists(origPath) = True Then
                    Dim pathOnly As String = Replace(origPath, TextBox1.Text, "")
                    Dim l As Integer = InStrRev(pathOnly, "\")

                    ' Create the path - path must be all lowercase (which it is already)
                    Dim newPath As String = Mid(pathOnly, 2, l - 2)
                    If InStr(newPath, "\") = 0 Then
                        Try
                            asu.CreateDirectory(yourFileShare, Prefix & "\" & newPath)
                        Catch ex As Exception

                        End Try
                    Else
                        Dim folders() As String = Split(newPath, "\")
                        Dim j As Integer
                        Dim catPath As String = ""
                        For j = 0 To folders.Count - 1
                            If j = 0 Then
                                catPath = folders(j)
                            Else
                                catPath = catPath & "\" & folders(j)
                            End If
                            Try
                                asu.CreateDirectory(yourFileShare, Prefix & "\" & catPath)
                            Catch ex As Exception

                            End Try
                        Next
                    End If

                    newPath = Prefix & "/" & Replace(newPath, "\", "/")

                    Dim dt As Date = DateAndTime.Now
                    sw2.WriteLine("Attempting " & origPath & " at " & dt.ToLongTimeString & " on " & dt.ToShortDateString)
                    tsData1.Text = "#" & Trim(Str(i + 1)) & " of " & Trim(Str(txtFilesArray.Length))

                    Try
                        asu.UploadFileAsFileToAzureStorage(yourFileShare, newPath, origPath, asu.JobOnly(origPath) & Path.GetExtension(origPath))
                        sw2.WriteLine("Uploading ..." & origPath & "...success.")
                    Catch ex As Exception
                        sw2.WriteLine("Uploading ..." & origPath & "...failed.")
                    End Try

                    sw2.WriteLine(" ")
                End If
            Next

            mWorking = False
        End While

        sw2.Close()
        sw2.Dispose()
        sw2 = Nothing

        WaitCursor(False)

        MsgBox("Done!")
    End Sub

    Public Sub WaitCursor(ByVal ShowHourglass As Boolean)
        If ShowHourglass = True Then
            Me.Cursor = Cursors.WaitCursor
        Else
            Me.Cursor = Cursors.Default
        End If
    End Sub

    Public Function GetAppPath() As String
        'Obtain the applications's path
        GetAppPath = System.IO.Path.GetDirectoryName(Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase.ToString())

        GetAppPath = Replace(GetAppPath, "file:\", "")
    End Function

    Private Sub Button3_Click(sender As Object, e As EventArgs) Handles Button3.Click
        mWorking = False
    End Sub

    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load

    End Sub
End Class

Step 6: Run the Project

Let's say you want to clone all the files and sub-directories that's on "D:\CRM". To do this, run the project (F5) in Debug mode. Then,

1. Click the "Select Folder" button. fbd1, which you added as a folderBrowserDialog control, will now open up and allow you to browse for the folder that you want to have cloned. In this case, you'll navigate to the "D:\CRM" folder and click "OK". TextBox1 will now display your folder selection D:\CRM.
2. Next, click the "Upload to Cloud" button. The code will now enumerate all the files and sub-directories within the chosen root folder. Note, however, that any actual FILES within the root directory, WILL NOT BE ENUMERATED.
3. A message box will pop up, once the files have all been enumerated. This can take a while, if you happen to have thousands of files and sub-directories. Click OK, and the file/directory cloning will begin!
4. Included in the code is a StreamWriter that will create and write to a log file, detailing each file to be transferred and the result of each file transfer (success or failure). The log file is located in your app's run path, which will be bin\debug of your application.
5. Also, during the transfer, you'll see an update of each file # (i.e. #35 of 512) as they're being processed and uploaded. This is thanks to the added StatusStrip/ToolStripLabel called tsData1.
6. Click the Cancel button if you need to stop the transfer.
7. Note that if, for some reason, you need to start over, you can wipe out the entire File Share using your Azure portal. Just be careful not to wipe out the wrong file share! But that's the easiest way to "start over", because you cannot delete a folder unless it is empty, when using the portal.
8. Microsoft offers a free download called "Microsoft Azure Storage Explorer", which is a PC app that is very useful for dealing with your Azure Storage Account.
9. In the comment section of the form's code, I mention changing the ".png" to "." in order to search for all files. But the code is already set to using ".". However, if you're only looking for a specific extension, then go ahead and replace the "." with ".[desired extension]". This will then enumerate only those files, and not all the files.