CODEWITHSUNDEEP
hadoopamazon-web-servicesamazon-s3hdfs
RAbraham
RAbraham

Reputation: 6324

Hadoop distcp copy from S3: Signature does not match error

I am trying to copy a file from S3 to my hadoop HDFS on Amazon EC2.

The command that I am using is:

bin/hadoop distcp s3://<awsAccessKeyId>:<awsSecretAccessKey>@<bucket_name>/f1 hdfs://user/root/

Error that I get is:SignatureDoesNotMatch

org.jets3t.service.S3ServiceException: S3 GET failed for '/%2Ff1'

<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

<StringToSignBytes>...</StringToSignBytes>

<RequestId>...</RequestId>

<HostId>..</HostId>

<SignatureProvided>NsefW5en6P728cc9llkFIk6yGc4=\
    </SignatureProvided>

<StringToSign>GETMon, 05 Aug 2013 15:28:21 GMT/<bucket_name>/%2Ff1</StringToSign>

<AWSAccessKeyId><MY_ACCESS_ID><\ /AWSAccessKeyId></Error>

I have only one AWS Access Key Id and secret Key. I checked my AWS account and they are the same. I use the same AWS Access Key and secret Key to log on to my EC2 cluster. I have also tried using core-site.xml but that has not helped either.

Thanks, Rajiv

Upvotes: 1

Views: 4241

Answers (2)

Garren S
Garren S

Reputation: 5792

An alternative to regenerating the key that worked for me was to use -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey= flags when running distcp.

Example: hadoop distcp -Dfs.s3n.awsAccessKeyId= -Dfs.s3n.awsSecretAccessKey= s3n://path/to/log/dir hdfs://hdfs-node:8020/logs/

Note the use of s3n, which has a 5GB file limitation: Difference between Amazon S3 and S3n in Hadoop

Edit: Do not url-encode the secret access key, so slashes "/" and pluses "+" should be passed as they are!

Upvotes: 1

RAbraham
RAbraham

Reputation: 6324

Regenerating my AWS Key and Secret such that there is no forward slash in my secret worked for me. Ref: https://issues.apache.org/jira/browse/HADOOP-3733

Upvotes: 1

Related Questions