CODEWITHSUNDEEP
hadoopamazon-web-services
Tommy
Tommy

Reputation: 13642

AWS Spark Cluster setup errors

I have created an AWS keypair.

I am following the instructions here word for word: https://aws.amazon.com/articles/4926593393724923

When I type in "aws emr create-cluster --name SparkCluster --ami-version 3.2 --instance-type m3.xlarge --instance-count 3 --ec2-attributes KeyName=MYKEY --applications Name=Hive --bootstrap-actions Path=s3://support.elasticmapreduce/spark/install-spark"

replacing MYKEY with both the full path and just the name of my key pair (I've tried everything), I get the following error:

`A client error (InvalidSignatureException) occurred when calling the RunJobFlow operation: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

The Canonical String for this request should have been 'POST /

content-type:application/x-amz-json-1.1 host:elasticmapreduce.us-east-1.amazonaws.com user-agent:aws-cli/1.7.5 Python/2.7.8 Darwin/14.1.0 x-amz-date:20150210T180927Z x-amz-target:ElasticMapReduce.RunJobFlow

content-type;host;user-agent;x-amz-date;x-amz-target dbb58908194fa8deb722fdf65ccd713807257deac18087025cec9a5e0d73c572'

The String-to-Sign should have been 'AWS4-HMAC-SHA256 20150210T180927Z 20150210/us-east-1/elasticmapreduce/aws4_request c83894ad3b43c0657dac2c3ab7f53d384b956087bd18a3113873fceeabc4ae26'`

What am I doing wrong?

Upvotes: 1

Views: 1031

Answers (1)

Tommy
Tommy

Reputation: 13642

GOT IT. Sadly, the above page mentions nothing about having to set the environment variables AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY. You must do this first. I learned you had to do that first from a totally different setup guide: http://spark.apache.org/docs/1.2.0/ec2-scripts.html.

After I set that, the Amazon instructions worked.

Upvotes: 1

Related Questions