Can I create a Hive metastore using Presto?

Question

Basic question: Can I use Presto as the basis for my Hive metastore?

I am trying to install presto and hive on an amazon aws and I'm having trouble creating the Hive metastore. Presto seems to be installed and working properly apart from failing to connect to the metastore. The problem is that when I run Hive, it fails to create a metastore.

When I run Hive I get the following errors:

Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1449)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:63)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2661)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2680)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:425)
    ... 7 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1447)
    ... 12 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
        at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:358)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:73)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1447)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:63)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2661)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2680)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:425)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
        ... 20 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:404)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
    at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:73)
    ... 17 more

My conf/hive-site.xml is as follows:

<property>
<name>hive.metastore.uris</name>
<value>thrift://ec2..........amazonaws.com:8080</value>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore. </description>

<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:presto://ec2...........amazonaws.com:9083/hive;databaseName=metastore_db;create=true</value>
<description>JDBC connect string for a JDBC metastore</description>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>presto-jdbc-0.90.jar</value>
<description>Driver class name for a JDBC metastore</description>

Can I use presto to create my metastore or do I have to use mySQL or derby? If I can, I would greatly appreciate any help I can get solving this.

Thanks!

Answer 1

You may use the following steps when storing your metastore to MySQL:

CREATE DATABASE metastore_db;

CREATE USER 'hiveuser'@'%' IDENTIFIED BY 'hivepassword';

GRANT all on *.* to 'hiveuser'@localhost identified by 'hivepassword';

flush privileges;

Once done add the below to your hive-site.xml in $HIVE_HOME/conf/

<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://10.0.0.96/metastore_db?createDatabaseIfNotExist=true</value>
        <description>metadata is stored in a MySQL server</description>
        </property>

        <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>MySQL JDBC driver class</description>
        </property>

        <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hiveuser</value>
        <description>user name for connecting to mysql server </description>
        </property>

        <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>hivepassword</value>
        <description>password for connecting to mysql server </description>
        </property>

        <property>
        <name>hive.metastore.uris</name>
        <value>thrift://10.0.0.96:9083</value>
        <description>Hive metastore Thrift server</description>
        </property>

Then start the metastore service :

hive --service metastore &

Just comment if you want to know more or you get stuck at something.

Regards, Devi Prasad.

Answer 2

Thanks for the replies so far.

@Dain Sundstrom: I tried your suggestion of the bootstrap action, but had no real joy.

Having done more research into the hive structure, I don't think it is possible to use presto as I was trying. From what I understand now, presto uses hive as its database, but hive itself needs an underlying database of mySQL or derby which is where I was going wrong. So my plan is to now install mySQL, link hive to it, and do it that way. Step 2 of that plan is pray it works without too much trouble.

Thanks for your help.

Answer 3

There is a Presto bootstrap action for EMR that should handle all of the setup for you.

Answer 4

In theory you should be able to do this. But Presto needs a backend; which backed in Presto using for your metastore db? Whichever backend you use, it needs to support inserts.