Unable to add UDF in hive

Question

I have to add the following UDF in hive :

package com.hadoopbook.hive;

import org.apache.commons.lang.StringUtils;
import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

public class Strip extends UDF {
  private Text result = new Text();

  public Text evaluate(Text str) {
    if (str == null) {
      return null;
    }
    result.set(StringUtils.strip(str.toString()));
    return result;
  }

  public Text evaluate(Text str, String stripChars) {
    if (str == null) {
      return null;
    }
    result.set(StringUtils.strip(str.toString(), stripChars));
    return result;
  }
}

This is an example from the book "Hadoop : The definitive guide"

1. I created the .class file of above java file using the following command :

   hduser@nb-VPCEH35EN:~/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive$ javac Strip.java 
   

2. Then I created the jar file using the following command :

   hduser@nb-VPCEH35EN:~/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive$ jar cvf Strip.jar Strip Strip.class 
   Strip : no such file or directory
   added manifest
   adding: Strip.class(in = 915) (out= 457)(deflated 50%)
   

3. I added the geenrated jar file to hdfs directory with :

   hduser@nb-VPCEH35EN:~/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive$ hadoop dfs -copyFromLocal /home/hduser/Hadoop-tutorial/hadoop-book-master/ch17-hive/src/main/java/com/hadoopbook/hive/Strip.jar /user/hduser/input
   

4. I tried to create a UDf usign the following command :

   hive> create function strip as 'com.hadoopbook.hive.Strip' using jar 'hdfs://localhost/user/hduser/input/Strip.jar';
   

But I got an error as following :

converting to local hdfs://localhost/user/hduser/input/Strip.jar Added [/tmp/hduser_resources/Strip.jar] to class path Added resources: [hdfs://localhost/user/hduser/input/Strip.jar] Failed to register default.strip using class com.hadoopbook.hive.Strip FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask

5. I also tried to create temporary function. So I first added the jar file to hive using :

   hive> add jar hdfs://localhost/user/hduser/input/Strip.jar;
   converting to local hdfs://localhost/user/hduser/input/Strip.jar
   Added [/tmp/hduser_resources/Strip.jar] to class path
   Added resources: [hdfs://localhost/user/hduser/input/Strip.jar]
   

6. Then I tried to add the temporary function :

   hive> create temporary function strip as 'com.hadoopbook.hive.Strip';
   

But I got the following error :

FAILED: Class com.hadoopbook.hive.Strip not found FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask

The jar file was successully created and added to hive.Still it is showing that class not found. Can anyone please tell what is wrong with it ?

Answer 1

yes using IDE like eclipse is easy then making jar from CLI.

Creating jar file from command line you have to follow these steps:

First make project dirs under project dir ch17-hive:

• bin - will store .class (Strip.class) files
• lib - will store required external jars

• traget - will store jars that you will create

  [ch17-hive]$ mkdir bin lib traget
  [ch17-hive]$ ls
  bin  lib  src  target
  

copy required external jars to ch170hive/lib dir:

[ch17-hive]$ cp /usr/lib/hive/lib/hive-exec.jar lib/.
[ch17-hive]$ cp /usr/lib/hadoop/hadoop-common.jar lib/.

Now compile java from dir from which your class com.hadoopbook.hive.Strip resides, in your case its ch17-hive/src/main/java:

[java]$ pwd
/home/cloudera/ch17-hive/src/main/java
[java]$ javac  -d ../../../bin -classpath ../../../lib/hive-exec.jar:../../../lib/hadoop-common.jar com/hadoopbook/hive/Strip.java 

Create menifest file as:

[ch17-hive]$ cat MENIFEST.MF 
Main-Class: com.hadoopbook.hive.Strip
Class-Path: lib/hadoop-common.jar  lib/hive-exec.jar

Create jar as

[ch17-hive]$ jar cvfm target/strip.jar MENIFEST.MF -C bin .added manifest
adding: com/(in = 0) (out= 0)(stored 0%)
adding: com/hadoopbook/(in = 0) (out= 0)(stored 0%)
adding: com/hadoopbook/hive/(in = 0) (out= 0)(stored 0%)
adding: com/hadoopbook/hive/Strip.class(in = 915) (out= 456)(deflated 50%)

Now you project structure should look like:

[ch17-hive]$ ls *
MENIFEST.MF

bin:
com

lib:
hadoop-common.jar  hive-exec.jar

src:
main

target:
strip.jar

copy created jar to hdfs:

hadoop fs -put /home/cloudera/ch17-hive/target/strip.jar /user/cloudera/.

use it in HIVE:

hive> create function strip_new as 'com.hadoopbook.hive.Strip' using jar 'hdfs:/user/cloudera/strip.jar';
converting to local hdfs:/user/cloudera/strip.jar
Added [/tmp/05a13d23-8051-431f-a354-793abac66160_resources/strip.jar] to class path
Added resources: [hdfs:/user/cloudera/strip.jar]
OK
Time taken: 0.071 seconds
hive>