Reputation: 19
i am running a simple spark-submit job, e.g.:
enter code here spark-submit --class com.x.y.z.logan
/home/test/spark/sample.jar
table in jar file
hiveContext.sql("CREATE TABLE IF NOT EXISTS
databasename.tablename(es_column_name STRING) STORED BY
'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'index_name/log','es.mapping.names'
='tablecoulmname :es_column_name ', 'es.nodes' =
'192.168.x.1y:9200','es.input.json' = 'false',
'es.index.read.missing.as.empty' = 'yes' ,'es.index.auto.create' =
'yes') ")
hiveContext.sql("INSERT INTO TABLE test.incompleterun SELECT
s.streamname FROM incomplete s");
**ERROR**
client token: N/A
diagnostics: User class threw exception: java.lang.RuntimeException:
java.lang.RuntimeException: class
org.elasticsearch.hadoop.mr.EsOutputFormat$EsOutputCommitter not
org.apache.hadoop.mapred.OutputCommitter
ApplicationMaster host: 192.168.x.y
ApplicationMaster RPC port: 0
queue: root.users.test
start time: 1485286033939
final status: FAILED
tracking URL: some URL
user: test
Exception in thread "main" org.apache.spark.SparkException:
Application application_1485258812942_0008 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1035)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1082)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
ES-hadoop is working fine when we are creating external table in Hive interface and loading data into external table from hive table. it is not working when we include the same query in jar. the same jar file works fine when we are creating normal hive table. The problem here is showing the below error when we include external table in jar file CAN ANYONE HELP ME HOW TO FIX THIS ?
Upvotes: 1
Views: 419