CODEWITHSUNDEEP
hadoopazure-hdinsight
Rajesh
Rajesh

Reputation: 51

Hadoop HDInsight .NET SDK APIs to submit job

I am using HDInsight .NET Hadoop APIs to submit a Map Reduce job in an asp.net Application.

using Microsoft.Hadoop.Mapreduce;

var hadoop = Hadoop.Connect();

var result = hadoop.MapReduceJob.ExecuteJob ();

//also tried this, but same exception

//var result = hadoop.MapReduceJob.ExecuteJob(config);

The ExecuteJob() call fails and throws exceptions at run time. has anybody in this world been able to run this call successfully. Is it possible to customize Map() function by adding more input parameters or objects (other than just given in MapperBase class by Microsoft)? Can the logic inside Mapper and Reducer Methods access cache/database?

Upvotes: 3

Views: 968

Answers (1)

Jonathan Gao
Jonathan Gao

Reputation: 679

A sample of submitting a MapReduce job using HDInsight .NET SDK is posted here:

http://www.windowsazure.com/en-us/manage/services/hdinsight/submit-hadoop-jobs-programmatically/#mapreduce-sdk

// Define the MapReduce job
MapReduceJobCreateParameters mrJobDefinition = new MapReduceJobCreateParameters()
{
    JarFile = "wasb:///example/jars/hadoop-examples.jar",
    ClassName = "wordcount"
};

mrJobDefinition.Arguments.Add("wasb:///example/data/gutenberg/davinci.txt");
mrJobDefinition.Arguments.Add("wasb:///example/data/WordCountOutput");

// Get the certificate object from certificate store using the friendly name to identify it
X509Store store = new X509Store();
store.Open(OpenFlags.ReadOnly);
X509Certificate2 cert = store.Certificates.Cast<X509Certificate2>().First(item => item.FriendlyName == certfrientlyname);
JobSubmissionCertificateCredential creds = new JobSubmissionCertificateCredential(new Guid(subscriptionID), cert, clusterName);

// Create a hadoop client to connect to HDInsight
var jobClient = JobSubmissionClientFactory.Connect(creds);

// Run the MapReduce job
JobCreationResults mrJobResults = jobClient.CreateMapReduceJob(mrJobDefinition);

// Wait for the job to complete
WaitForJobCompletion(mrJobResults, jobClient);

Upvotes: 1

Related Questions