CODEWITHSUNDEEP
hive
harel
harel

Reputation: 525

Hive 13.0 The UDF implementation class '...' is not present in the class path

I encounter weird behaviour when using Hive 13.0.1 on Amazon EMR. This happens when I try to both use UDF and run external shell script that runs hive -e "..." commands

We have been using shell scripts to add partitions dynamically to a table and never encountered any problem in Hive 0.11

However in Hive 0.13.1 the following simplified example breaks:

add jar myjar;
create temporary function myfunc as '...';
create external table mytable...
!hive -e "";
select myfunc(someCol) from mytable;

Results in The UDF implementation class '...' is not present in the class path

Removing the shell command (!hive -e "") and the error disappears. Adding the jar and function again after the shell and the error disappears (Adding just the function without the jar does not get rid of the error).

Is this known behavior or a bug, can I do anything besides reloading the jar and function before every usage?

Upvotes: 1

Views: 424

Answers (1)

Joydeep Sen Sarma
Joydeep Sen Sarma

Reputation: 1214

AFAIK - this is the way it's always been. One hive shell cannot pass on the additional jars added to it's classpath to the child shell. and definitely not the function definitions.

We provide Hive/Hadoop etc. as a service in Qubole and have the notion of a hive bootstrap that is used to, for cases like this, capture common statements required for all queries. This is used extensively by most users. (caveat - i am one of Qubole and Hive's founders - but would recommend using Qubole over EMR for Hive).

Upvotes: 1

Related Questions