Oozie Hive workflow intermitent

Question

I currently have a problem where intermittently my oozie workflow will be unable to connect to my hive metastore. It seems like it is running out of connections to the hive-metastore?

Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused at org.apache.thrift.transport.TSocket.open(TSocket.java:185) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:277) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:163) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1082) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:51) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:61) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2140) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2151) at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:1013) at org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:1000) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:8732) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8097) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:443) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:908) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:347) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:445) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:455) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:711) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:261) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:238) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:49) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:491) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at org.apache.thrift.transport.TSocket.open(TSocket.java:180) ... 44 more ) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:323) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:163) ... 42 more FAILED: Error in metadata: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

I can run hive from the command line. Use hue to communicate to the hive-metastore and execute queries. This only seems to happen half way through my oozie workflows. Each and every hive action i commit has the hive-site.xml config as oozie.hive.defaults.

hive-site.xml

 <property>   <name>hive.metastore.uris</name>  
 <value>thrift://localhost:9083</value>   <description>Thrift uri for
 the remote metastore. Used by metastore client to connect to remote
 metastore.</description> </property>

 <property>   <name>datanucleus.fixedDatastore</name>
     <value>false</value> </property>

 <property>   <name>javax.jdo.option.ConnectionURL</name>  
 <value>jdbc:mysql://localhost:3306/metastore</value>  
 <description>JDBC connect string for a JDBC metastore</description>
 </property>

 <property>   <name>javax.jdo.option.ConnectionDriverName</name>  
 <value>com.mysql.jdbc.Driver</value>   <description>Driver class name
 for a JDBC metastore</description> </property>

 <property>   <name>javax.jdo.option.ConnectionUserName</name>  
 <value>hive</value>   <description>username to use against metastore
 database</description> </property>

 <property>   <name>javax.jdo.option.ConnectionPassword</name>  
 <value>hive</value>   <description>password to use against metastore
 database</description> </property>

hive-metastore, hive-server2, mysql-server and oozie all run on the same host for the moment so localhost works. Any ideas? I have oozie share lib default true enabled and the sharelibs has been created.

CDH 4.2.1 packages

-oozie: oozie-3.3.0

-hive-metastore: hive-metastore-0.10.0

-hive-server2: hive-server2-0.10.0

-mysql-server: mysql-server-5.1.69-1

Any help would be greatly appreciated

Answer 1

oozie.hive.defaults is deprecated in the Hive action. Can you try to have Job Xml pointing to the hive-site.xml uploaded on the HDFS?