CODEWITHSUNDEEP
javaamazon-web-servicesemr
SQL.injection
SQL.injection

Reputation: 2647

where is my EMR cluster

I am trying to create an EMR cluster on java, but i can't neither find it on the EMR cluster list, neither can see the instances requested on EC2.

EMR roles do exist:

sqlInjection@VirtualBox:~$ aws iam list-roles | grep EMR
            "RoleName": "EMR_DefaultRole", 
            "Arn": "arn:aws:iam::removed:role/EMR_DefaultRole"
            "RoleName": "EMR_EC2_DefaultRole", 
            "Arn": "arn:aws:iam::removed:role/EMR_EC2_DefaultRole"

and now my java code:

AWSCredentials awsCredentials = new BasicAWSCredentials(awsKey, awsKeySecret);
AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(awsCredentials);

StepFactory stepFactory = new StepFactory();
        StepConfig enabledebugging = new StepConfig()
               .withName("Enable debugging")
               .withActionOnFailure("TERMINATE_JOB_FLOW")
               .withHadoopJarStep(stepFactory.newEnableDebuggingStep());

    HadoopJarStepConfig hadoopConfig1 = new HadoopJarStepConfig()
    .withJar("s3://foo.bucket/hadoop_jar/2015-01-12/foo.jar")
    .withMainClass("com.strackoverflow.DriverFoo") // optional main class, this can be omitted if jar above has a manifest
    .withArgs("--input=s3://foo.bucket/logs/,s3://foo.bucket/morelogs/", "--output=s3://foo.bucket/myEMROutput" , "--inputType=text"); // i have custom java code to handle the --input, --output and --inputType parameters


    StepConfig customStep = new StepConfig("Step1", hadoopConfig1);

    Collection <StepConfig> steps = new ArrayList<StepConfig>();
    {
        steps.add(enabledebugging);
        steps.add(customStep);
    }


    JobFlowInstancesConfig instancesConfig = new JobFlowInstancesConfig()
    .withEc2KeyName("fookey") //not fookey.pem
    .withInstanceCount(2)
    .withKeepJobFlowAliveWhenNoSteps(false) // on aws example is set to true
    .withMasterInstanceType("m1.medium")
    .withSlaveInstanceType("m1.medium");


    RunJobFlowRequest request = new RunJobFlowRequest()
     .withName("java programatic request")
     .withAmiVersion("3.3.1")
     .withSteps(steps) // on the amazon example is lunched debug and hive, here is debug and a jar
     .withLogUri("s3://devel.rui/emr_clusters/pr01/")
     .withInstances(instancesConfig)
     .withVisibleToAllUsers(true);

       RunJobFlowResult result = emr.runJobFlow(request);

       System.out.println("toString "+ result.toString());
       System.out.println("getJobFlowId "+ result.getJobFlowId());
       System.out.println("hashCode "+ result.hashCode());

Where is my cluster? I cannot see it on cluster list, output folder is not created, logs folder stays empty and no instances are visible on EC2.

by the program outputs this

toString {JobFlowId: j-2xxxxxxU}
getJobFlowId j-2xxxxxU
hashCode -1xxxxx4

I had follow the instruction from here to create the cluster http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/calling-emr-with-java-sdk.html

And this to create the java job http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-common-programming-sample.html

Upvotes: 1

Views: 1083

Answers (1)

SQL.injection
SQL.injection

Reputation: 2647

On the Amazon example, the region is not configured.

After configuring the region the cluster is launched properly.

    AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(awsCredentials);
    emr.setRegion(Region.getRegion(Regions.EU_WEST_1));

Upvotes: 1

Related Questions