CODEWITHSUNDEEP
apache-sparkhadoophiveapache-spark-sqlhiveql
Avinash
Avinash

Reputation: 83

Spark SQL and using existing hive udfs

I have to read existing hive udf in spark, so in spark.sql I am creating temporary function and using it, It is working fine in spark shell but in spark-submit it's failing with the error:

org.apache.hadoop.hive.metadata.HiveException : unable to execute method public static boolean com.xxx.x.x.udfs.isb_udf.evaluate(Java.lang.string) on object xxxx.udf of class xxxx with args {Java.lang.string} of size 1 ...

I also see... Caused by Java.lang.reflect.InvocationTargetException

Code sample:

spark.sql("CREATE TEMPORARY FUNCTION currency AS 'com.spark.udf.FormatCurrency'"); 
val x = spark.sql("select currency(col1) from hive_table") ;

x.show()

Above command works in spark shell but it is not working in spark submit.

Info: If I read hive table without udf it's working in spark submit. The problem probably occurs when using hive udf.

Upvotes: 2

Views: 1095

Answers (1)

Iam Rey
Iam Rey

Reputation: 31

One of the reason would be, if you’re using multiple jars passing to spark submit and it contains the same class that your UDF is using then the issue might occur. I have a similar case before and I removed the jars and added a single jar in —jars along with spark submit. It works for me. I don’t know about your case. Try removing multiple jars and for sure it will work.

Upvotes: 1

Related Questions