CODEWITHSUNDEEP
aws-cli
sam
sam

Reputation: 85

unable to create aws emr cluster with enable-debugging

I am not able to create aws emr cluster when I add command "--enable-debugging" I am able to create the cluster without enable-debugging command. Getting error like: aws: error: invalid json argument for option --configurations

my script to create cluster is :

aws emr create-cluster \
  --name test-cluster \
  --release-label emr-5.5.0 \
  --instance-groups     InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m3.xlarge InstanceGroupType=CORE,InstanceCount=1,InstanceType=m3.xlarge \
  --no-auto-terminate \
  --termination-protected \
  --visible-to-all-users \
  --use-default-roles \
  --log-uri s3://testlogs/ \
  --enable-debugging \
  --tags Owner=${OWNER} Environment=Dev Name=${OWNER}-test-cluster \
  --ec2-attributes KeyName=$KEY,SubnetId=$SUBNET \
  --applications Name=Hadoop Name=Pig Name=Hive \
  --security-configuration test-sec-config \
  --configurations s3://configurations/mapreduceconfig.json

mapreduceconfig.json file is :

[
 {
   "Classification": "mapred-site",
   "Properties": {
       "mapred.tasktracker.map.tasks.maximum": 2
   }
 },
 {
   "Classification": "hadoop-env",
   "Properties": {},
   "Configurations": [
       {
         "Classification": "export",
         "Properties": {
             "HADOOP_DATANODE_HEAPSIZE": 2048,
             "HADOOP_NAMENODE_OPTS": "-XX:GCTimeRatio=19"
         }
       }
   ]
 }
]

Upvotes: 1

Views: 955

Answers (1)

jc mannem
jc mannem

Reputation: 2343

Well, the error is self apparent. --configurations option does not support S3:// file system. As per the examples and documentation http://docs.aws.amazon.com/cli/latest/reference/emr/create-cluster.html

It only supports file:// and a direct public link to a file in S3. like https://s3.amazonaws.com/myBucket/mapreduceconfig.json

So, your configurations gotta be Public.

Not sure how you got it working without --enable-debugging command.

Upvotes: 2

Related Questions