CODEWITHSUNDEEP
c#restapiazurehadoop
Saravanakumar Natarajan
Saravanakumar Natarajan

Reputation: 445

How to Append a File using Web HDFS REST API in C#?

I am using Azure Data Lake to upload the file before but still want to append the text file content existing Data Lake text file. Is there any option available to append the text file data using Web HDFS REST API in C#?.

I am refer this link enter link description here

Code: I Can refer the above link get the append URL. But how can i use this URL and Append a File using c#? 

private const string AppendUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?&op=APPEND&noredirect=true";

Upvotes: 1

Views: 1021

Answers (1)

Tom Sun
Tom Sun

Reputation: 24549

If you want to use the Rest Api to do that we could use the following code. I test it with Postman.

private const string AppendUrl = "https://{datalakeName}.azuredatalakestore.net/webhdfs/v1/{filepath}?append=true&op=APPEND&api-version=2016-11-01"
var token = "eyJ0eX.....";
using (var client = new HttpClient())
{
     client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
     var result = client.GetAsync(url).Result;
     var data = result.Content.ReadAsStringAsync().Result;
}

enter image description here

We also could use the Azure Microsoft.Azure.Management.DataLake.Store to do that. How to get the application id and secretkey you could refer to official document. More detail steps to get the permission to access the datalake you could refer to another SO thread.

var applicationId = "application Id";
var secretKey = "secretKey";
var tenantId = "tenant id";
var adlsAccountName = "datalake account name";
var creds = ApplicationTokenProvider.LoginSilentAsync(tenantId, applicationId, secretKey).Result;
var adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds,clientTimeoutInMinutes:60);
var stream = File.OpenRead(@"C:\tom\testtext.txt");
var test = adlsFileSystemClient.FileSystem.AppendWithHttpMessagesAsync(adlsAccountName, "test/abc.txt", stream).Result;

enter image description here

packages:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.Management.DataLake.Store" version="2.3.0-preview" targetFramework="net452" />
  <package id="Microsoft.Azure.Management.DataLake.StoreUploader" version="1.0.0-preview" targetFramework="net452" />
  <package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.13.8" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime" version="2.3.9" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime.Azure" version="3.3.9" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime.Azure.Authentication" version="2.2.0-preview" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="9.0.2-beta1" targetFramework="net452" />
</packages>

Upvotes: 2

Related Questions