CODEWITHSUNDEEP
amazon-web-servicesaws-sdkemr
autodidacticon
autodidacticon

Reputation: 1358

AWS EMRFS Consistent View enable via SDK

Typically emrfs consistency is enabled via emrfs-site.xml http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emrfs-configure-consistent-view.html

Does anyone know if these setting can be accessed via the SDK?

Upvotes: 0

Views: 1087

Answers (2)

Rori Stumpf
Rori Stumpf

Reputation: 1977

To enable EMRFS with the Java SDK, an "emrfs-site" configuration needs to be added to the RunJobFlowRequest and the fs.s3.consistent property must be set to true. Like this:

Map<String, String> emrfsProperties = new HashMap<>();
emrfsProperties.put("fs.s3.consistent", "true");

RunJobFlowRequest request = new RunJobFlowRequest()
        ....
        .withServiceRole(SERVICE_ROLE)
        .withJobFlowRole(JOB_FLOW_ROLE)
        .withConfigurations(
                new Configuration().withClassification("yarn-site").withProperties(yarnProperties),
                new Configuration().withClassification("emrfs-site").withProperties(emrfsProperties)
        )
        .withInstances(new JobFlowInstancesConfig()
                .withEc2KeyName(EC2_KEYPAIR)
                ....

A full list of EMRFS configuration parameters can be found here

Upvotes: 3

m.aibin
m.aibin

Reputation: 3603

Yes, you have a full documentation here: http://docs.aws.amazon.com/ElasticMapReduce/latest/API/Welcome.html

You need to authorize connection to your AWS first, than you can configure you application to your needs, using API. Look also here: http://docs.aws.amazon.com/ElasticMapReduce/latest/API/CommonParameters.html http://docs.aws.amazon.com/ElasticMapReduce/latest/API/EmrConfigurations.html

Upvotes: 0

Related Questions