CODEWITHSUNDEEP
apache-sparkamazon-s3amazon-emr
lfk
lfk

Reputation: 2633

400 Bad Request while trying to access S3 from Spark

I'm trying to read a file thorugh s3a from a bucket in us-east-2 (Ohio) and I'm getting 400 Bad Request response:

com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: [removed], AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: [removed]

Since my code works with another bucket in Sydney, it seems to be a signing API version issue (Ohio supports only 4, Sydney supports 2 and 4). So I tried setting the endpoint by adding this to spark-submit as suggested in other posts:

--conf "spark.hadoop.fs.s3a.endpoint=s3.us-east-2.amazonaws.com"

But that didn't make any difference. I also tried adding the same to a conf file and passing it using --properties-file [file_path]

spark.hadoop.fs.s3a.endpoint                    s3.us-east-2.amazonaws.com

No difference. I still get the same error for Ohio (and it doesn't work with Sydney any more, for obvious reasons).

Upvotes: 2

Views: 4395

Answers (1)

siddhu salvi
siddhu salvi

Reputation: 59

Add the following line in your code:

System.setProperty("com.amazonaws.services.s3.enableV4", "true")

Upvotes: 2

Related Questions