CODEWITHSUNDEEP
amazon-s3apache-sparkhivehdfs
tribbloid
tribbloid

Reputation: 3858

Setting AWS credentials on Spark program using 3 methods, None of them works

I'm launching a Spark hive-server cluster that use S3 as a warehouse. I've redundantly set up my AWS credential using 3 methods, namely:

  1. hdfs-site.xml under $SPARK_HOME/conf:

<property>
  <name>fs.s3.awsAccessKeyId</name>
  <value>****</value>
</property>

<property>
  <name>fs.s3.awsSecretAccessKey</name>
  <value>****</value>
</property>

  1. setting executor's system property by using spark.executor.extraJavaOptions in start-hivethrift parameter:

    --conf "spark.executor.extraJavaOptions=-Dfs.s3.awsAccessKeyId=**** -Dfs.s3.awsSecretAccessKey=****" \

  2. setting environment variables before start-hivethrift.

The launching script looks like this:

AWS_ACCESS_KEY_ID=**** \
AWS_SECRET_ACCESS_KEY=**** \
$SPARK_HOME/sbin/start-thriftserver.sh \
--conf "spark.executor.extraJavaOptions=-Dfs.s3.awsAccessKeyId=**** -Dfs.s3.awsSecretAccessKey=****" \
--hiveconf hive.metastore.warehouse.dir=s3n://testdata \

but when I run any create table query I still get:

Error: org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively).) (state=,code=0)

What is going on here? Why none of them works as in documentation?

Upvotes: 2

Views: 2530

Answers (1)

tribbloid
tribbloid

Reputation: 3858

Oops, there is a problem in my hdfs-site.xml. I should add all possible schema names supported by S3:

<configuration>

<property>
  <name>fs.s3.awsAccessKeyId</name>
  <value>****</value>
</property>

<property>
  <name>fs.s3.awsSecretAccessKey</name>
  <value>****</value>
</property>

<property>
  <name>fs.s3n.awsAccessKeyId</name>
  <value>****</value>
</property>

<property>
  <name>fs.s3n.awsSecretAccessKey</name>
  <value>****</value>
</property>

<property>
  <name>fs.s3a.awsAccessKeyId</name>
  <value>****</value>
</property>

<property>
  <name>fs.s3a.awsSecretAccessKey</name>
  <value>****</value>
</property>

</configuration>

Seems no more problem now. Its kind of inconvenient but I'm glad it works now.

Upvotes: 2

Related Questions