CODEWITHSUNDEEP
amazon-web-serviceshadoopapache-sparkencryptionamazon-s3
Vikash Pareek
Vikash Pareek

Reputation: 1181

Doesn't Spark/Hadoop support SSE-KMS encryption on AWS S3

I am trying to save an rdd on S3 with server side encryption using KMS key (SSE-KMS), But I am getting the following exception:

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 695E32175EBA568A, AWS Error Code: InvalidArgument, AWS Error Message: The encryption method specified is not supported, S3 Extended Request ID: Pi+HFLg0WsAWtkdI2S/xViOcRPMCi7zdHiaO5n1f7tiwpJe2z0lPY1C2Cr53PnnUCj3358Gx3AQ=

Following is the piece of my test code to write an rdd on S3 by using SSE-KMS for encryption:

val sparkConf = new SparkConf().
  setMaster("local[*]").
  setAppName("aws-encryption")
val sc = new SparkContext(sparkConf)

sc.hadoopConfiguration.set("fs.s3a.access.key", AWS_ACCESS_KEY)
sc.hadoopConfiguration.set("fs.s3a.secret.key", AWS_SECRET_KEY)
sc.hadoopConfiguration.setBoolean("fs.s3a.sse.enabled", true)
sc.hadoopConfiguration.set("fs.s3a.server-side-encryption-algorithm", "SSE-KMS")
sc.hadoopConfiguration.set("fs.s3a.sse.kms.keyId", KMS_ID)

val s3a = new org.apache.hadoop.fs.s3a.S3AFileSystem
val s3aName = s3a.getClass.getName
sc.hadoopConfiguration.set("fs.s3a.impl", s3aName)

val rdd = sc.parallelize(Seq("one", "two", "three", "four"))
println("rdd is: " + rdd.collect())
rdd.saveAsTextFile(s"s3a://$bucket/$objKey")

Although, I am able to write rdd on s3 with AES256 encryption.

Does spark/hadoop have a different value for KMS key encryption instead of "SSE-KMS"?

Can anyone please suggest what I am missing here or doing wrong.

Environment details as follow:

Thank you in advance.

Upvotes: 2

Views: 4826

Answers (1)

Vikash Pareek
Vikash Pareek

Reputation: 1181

Unfortunately, It seems like existing version of Hadoop i.e. 2.8 does not support SSE-KMS :(

Following is the observation:

  1. SSE-KMS is not supported till Hadoop 2.8.1
  2. SSE-KMS supposed to be introduced in Hadoop 2.9
  3. In Hadoop 3.0.0alpha version, SSE-KMS is supported.

Same observation w.r.t. AWS SDK for Java

  1. SSE-KMS was introduced in aws-java-sdk 1.9.5

Upvotes: 1

Related Questions