Unable to connect to HIVE java

Question

I want to use org.apache.hadoop.hive to work with metastore.

The hive(1.1) and hadoop(version 2.6 ) was installed on a linux server. My computer has a windows OS. Here, I am trying to create Hive conf.

import org.apache.hadoop.hive.conf.HiveConf;


    public class Main {
        public static void main(String[] args){

            HiveConf hiveConf = new HiveConf();
            hiveConf.setIntVar(HiveConf.ConfVars.METASTORETHRIFTCONNECTIONRETRIES, 3);
            hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, "thrift://server:port");

            HiveMetastore hiveMetaStoreConnector = new HiveMetastore(hiveConf);
            if(hiveMetaStoreConnector != null){
                System.out.print(hiveMetaStoreConnector.getAllPartitionInfo("tablename"));
            }
        }
    }

But I encountered a problem in HiveConf hiveConf = new HiveConf();

SEVERE: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. at
org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379) at
org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394) at org.apache.hadoop.util.Shell.(Shell.java:387) at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2065)
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:332)
at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:95) at Main.main(Main.java:11) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:95) at Main.main(Main.java:11) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) Caused by: java.lang.RuntimeException: Could not load shims in class null
at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:86)
at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:62) at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:335) ... 7 more Caused by: java.lang.NullPointerException at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:83) ... 9 more

Should I install the hadoop client on my windows(winutils.exe)? Or should I include more libraries?

Also, I need to set conf only hiveConf.setVar(HiveConf.ConfVars.METASTOREURIS, "thrift://server:port")?

Or is it something to do with setting the smth?

Any suggestions will be appreciated.

Answer 1

I am not sure what you are trying to achieve, but your code would not run on the windows machine, because the windows machine is missing hadoop library. I think it is not possible to have a true setup on bare machine. You would need some kind of VMware software to get the window machine work as hive client.

If you are trying to connect to hive through a program, you should try using Hive API, which exposes JDBC way to connect to hive.

JDBC

HiveServer2 has a JDBC driver. It supports both embedded and remote access to HiveServer2. Remote HiveServer2 mode is recommended for production use, as it is more secure and doesn't require direct HDFS/metastore access to be granted for users.

The HiveServer2 URL is a string with the following syntax:

jdbc:hive2://<host1>:<port1>,<host2>:<port2>/dbName;initFile=<file>;sess_var_list?hive_conf_list#hive_var_list

where

<host1>:<port1>,<host2>:<port2> is a server instance or a comma separated list of server instances to connect to (if dynamic service discovery is enabled). If empty, the embedded server will be used.

dbName is the name of the initial database.

<file> is the path of init script file (Hive 2.2.0 and later). This script file is written with SQL statements which will be executed automatically after connection. This option can be empty.

sess_var_list is a semicolon separated list of key=value pairs of session variables (e.g., user=foo;password=bar).

hive_conf_list is a semicolon separated list of key=value pairs of Hive configuration variables for this session

hive_var_list is a semicolon separated list of key=value pairs of Hive variables for this session.