CODEWITHSUNDEEP
c#sftpazure-blob-storage
Mukil Deepthi
Mukil Deepthi

Reputation: 6452

Traverse files in sftp and store to azure blob

Basically i creating a azure webjob which will loop through record from a sql table which has got sftp location detail. for each sftp location, connect using sftpclient and read all files in a folder. for each files, i connect to azure and store it in the blob.

I have done the above in single file. But wish to do in a proper object oriented way. Not sure what is the right approach. i am not very much experience in design pattern but would love if someone can recomment a right approach.

Appreciate if someone can help me to achieve this in a proper object oriented way.

Thanks

            var azureBlob = AzureBlobStorage.Instance(StorageConnectionString, Containername);

            List<SftpLocationData> sftps = null;

            try
            {
                sftps = SftpLocationClient.GetSftpLocationDetails().ToList();
                if (sftps == null || !sftps.Any()) return;

            }
            catch (Exception ex)
            {
                LogMessage(string.Format("Error getting sftp details : {0}", ex.Message), log);
            }

            foreach (var fileClient in sftps.Select(sftp => new FileTransferClient(sftp)))
            {
                using (var sftpClient = fileClient.CreateClient())
                {
                    sftpClient.Connect();

                    var files = sftpClient.ListDirectory(path: fileClient.Data.Directory ?? ".").ToList();
                    if (files.Any())
                    {
                        var validFiles = files.Where(f => ext.Any(e => e == Path.GetExtension(f.Name))).ToList();
                        foreach (var file in validFiles)
                        {
                            var fileExists = azureBlob.FileExists(file.Name);
                            var blobUri = string.Empty;
                            var blobName = file.Name;
                            var fileImport = PopulateFileImportData(file);
                            if (fileExists)
                            {
                                int count = azureBlob.ListFiles(blobName);
                                blobName = (count == 0) ? blobName : String.Format("{0}_v{1}", blobName, count);
                                fileImport.Error = true;
                                fileImport.ErrorMsg = "Duplicate File";
                            }
                            fileImport.FileName = blobName;
                            try
                            {
                                var fileSaved = RbsClient.SaveFileImport(fileImport);

                                blobUri = azureBlob.UploadFile(fileSaved.FileName, sftpClient.OpenRead(file.FullName));

                                fileSaved.Archived = DateTime.Now;
                                RRClient.UpdateFile(fileSaved);
                            }
                            catch (Exception ex)
                            {
                                LogMessage(string.Format("Error saving fileimport detail : {0}", ex.Message), log);
                            }

                            if (fileImport.Error) continue;

                            IQueueGenerator queueGenerator = new QueueGenerator();

                            var queueName = queueGenerator.GetQueueName(
                                                blobName,
                                                fileClient.Data,
                                                blobUri);

                            if (string.IsNullOrEmpty(queueName)) continue;
                        }
                    }
                    sftpClient.Disconnect();
                }
            }
        }
        catch (Exception ex)
        {
            LogMessage(string.Format("Error occurred in processing pending altapay requests. Error : {0}", ex.Message), log);
        }

Upvotes: 1

Views: 2180

Answers (1)

Vasistan
Vasistan

Reputation: 292

Hi as of my understanding you need to copy the files/folders from FTP to azure blob storage.

Step 1. Get list of SFTP locations
Step 2. Init Azure Blob Storage Connection
Step 3. Init SFTP
Step 4. Iterate Files in SFTP
Step 5. Store the file into Blob Storage
Step 6. Close SFTP connection
Step 7. Close Azure Blob Storage

Upvotes: 1

Related Questions