CODEWITHSUNDEEP
c#azureazure-data-lakedata-lake
Vladimir Semashkin
Vladimir Semashkin

Reputation: 1280

Azure Data Lake: How to get Processed files

I've just started working with Data Lake and I'm currently trying to figure out the real workflow steps and how to automatize the whole process. Say I have some files as an input and I would like to process them and download output files in order to push into my data warehouse or/and SSAS.

I've found absolutely lovely API and it's all good but I can't find a way to get all the file names in a directory to get them downloaded further.

Please correct my thoughts regarding workflow. Is there another, more elegant way to automatically get all the processed data (outputs) into a storage (like conventional SQL Server, SSAS, data warehouse and etc)?

If you have a working solution based on Data Lake, please describe the workflow (from "raw" files to reports for end-users) with a few words.

here is my example of NET Core application

using Microsoft.Azure.DataLake.Store;
using Microsoft.IdentityModel.Clients.ActiveDirectory;
using Microsoft.Rest.Azure.Authentication;

            var creds = new ClientCredential(ApplicationId, Secret);
            var clientCreds = ApplicationTokenProvider.LoginSilentAsync(Tenant, creds).GetAwaiter().GetResult();
            var client = AdlsClient.CreateClient("myfirstdatalakeservice.azuredatalakestore.net", clientCreds);
            var result = client.GetDirectoryEntry("/mynewfolder", UserGroupRepresentation.ObjectID);

Upvotes: 1

Views: 546

Answers (1)

Tom Sun
Tom Sun

Reputation: 24529

Say I have some files as an input and I would like to process them and download output files in order to push into my data warehouse or/and SSAS.

If you want to download the files from the folder in the azure datalake to the local path, you could use the following code to do that.

client.BulkDownload("/mynewfolder", @"D:\Tom\xx"); //local path

But based on my understanding, you could use the azure datafactory to push your data from datalake store to azure storage blob or azure file storge.

Upvotes: 1

Related Questions