CODEWITHSUNDEEP
hadoophivehortonworks-data-platform
sio2deep
sio2deep

Reputation: 113

Select statement error with Application exitCode 1

Am working on Hortonworks Hive. I have seen same type of errors. But underlying MapReduce error seems to be different here in the case as Application error with exitCode 1.

In Hive, the statement

Select * from SomeTable;

...Is working fine, but

Select colName from SomeTable;

...Is not working.

Application error log

2014-03-17 12:49:15,557 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1395039411618_0001 State change from ACCEPTED to FAILED
2014-03-17 12:49:15,558 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application appattempt_1395039411618_0001_000002 is done. finalState=FAILED
2014-03-17 12:49:15,559 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1395039411618_0001 requests cleared
2014-03-17 12:49:15,559 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1395039411618_0001 user: asande queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2014-03-17 12:49:15,559 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application removed - appId: application_1395039411618_0001 user: asande leaf-queue of parent: root #applications: 0
2014-03-17 12:49:15,559 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=asande   OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE  DESCRIPTION=App failed with state: FAILED   PERMISSIONS=Application application_1395039411618_0001 failed 2 times due to AM Container for appattempt_1395039411618_0001_000002 exited with  exitCode: 1 due to: Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException:

Here's Hive.log. (But it seems there's nothing wrong in the log.)

<code>
2014-03-17 10:45:37,322 INFO  server.HiveServer2 (HiveStringUtils.java:startupShutdownMessage(604)) - STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting HiveServer2
STARTUP_MSG:   host = ASANDE1/16.155.82.203
STARTUP_MSG:   args = [-hiveconf, hive.hadoop.classpath=c:\hdp\hive-0.12.0.2.0.6.0-0009\lib\*.............................................., -hiveconf, hive.querylog.location=c:\hadoop\logs\hive\history, -hiveconf, hive.log.dir=c:\hadoop\logs\hive]
STARTUP_MSG:   version = 0.12.0.2.0.6.0-0009
STARTUP_MSG:   classpath = c:\hdp\hadoop-2.2.0.2.0.6.0-0009\etc\hadoop;c:\hdp\hadoop-;;
STARTUP_MSG:   build = git://sijenkins-vm3/cygdrive/d/w/bw/project/hive-monarch -r a7f54db5645b645500778b92e7fad8fab7738080; compiled by 'jenkins' on Fri Dec 20 18:29:58 PST 2013
************************************************************/
2014-03-17 10:45:39,622 INFO  service.CompositeService (SessionManager.java:init(60)) - HiveServer2: Async execution thread pool size: 100
2014-03-17 10:45:39,622 INFO  service.CompositeService (SessionManager.java:init(62)) - HiveServer2: Async execution wait queue size: 100
2014-03-17 10:45:39,623 INFO  service.CompositeService (SessionManager.java:init(64)) - HiveServer2: Async execution thread keepalive time: 10
2014-03-17 10:45:39,628 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited.
2014-03-17 10:45:39,628 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited.
2014-03-17 10:45:39,628 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited.
2014-03-17 10:45:39,628 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:ThriftBinaryCLIService is inited.
2014-03-17 10:45:39,628 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited.
2014-03-17 10:45:39,628 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started.
2014-03-17 10:45:39,628 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started.
2014-03-17 10:45:39,628 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started.
2014-03-17 10:45:39,925 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://ASANDE1:9083
2014-03-17 10:46:02,239 WARN  hive.metastore (HiveMetaStoreClient.java:open(307)) - set_ugi() not successful, Likely cause: new client talking to old server. Continuing without it.
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:2822)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:2808)

Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
    ... 14 more
2014-03-17 10:46:02,280 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2014-03-17 10:46:03,280 INFO  hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore.
2014-03-17 10:46:08,455 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(960)) - Got exception: org.apache.thrift.TApplicationException get_databases failed: out of sequence response
org.apache.thrift.TApplicationException: get_databases failed: out of sequence response
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:76)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_databases(ThriftHiveMetastore.java:500)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_databases(ThriftHiveMetastore.java:487)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:722)
    at org.apache.hive.service.cli.CLIService.start(CLIService.java:83)
    at org.apache.hive.service.CompositeService.start(CompositeService.java:70)
    at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:73)
    at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:103)
2014-03-17 10:46:08,455 ERROR hive.log (MetaStoreUtils.java:logAndThrowMetaException(961)) - Converting exception to MetaException
2014-03-17 10:46:08,621 ERROR service.CompositeService (CompositeService.java:start(74)) - Error starting services HiveServer2
org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
    at org.apache.hive.service.cli.CLIService.start(CLIService.java:85)
    ..........................
Caused by: MetaException(message:Got exception: org.apache.thrift.TApplicationException get_databases failed: out of sequence response)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:962)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:724)
    at org.apache.hive.service.cli.CLIService.start(CLIService.java:83)
    ... 3 more
2014-03-17 10:46:08,627 INFO  service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped.
2014-03-17 10:46:08,627 INFO  service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped.
2014-03-17 10:46:08,627 INFO  service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped.
2014-03-17 10:46:08,627 FATAL server.HiveServer2 (HiveServer2.java:main(105)) - Error starting HiveServer2
org.apache.hive.service.ServiceException: Failed to Start HiveServer2
    at org.apache.hive.service.CompositeService.start(CompositeService.java:80)
    at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:73)
    at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:103)
Caused by: org.apache.hive.service.ServiceException: Unable to connect to MetaStore!
    at org.apache.hive.service.cli.CLIService.start(CLIService.java:85)
    at org.apache.hive.service.CompositeService.start(CompositeService.java:70)
    ... 2 more
Caused by: MetaException(message:Got exception: org.apache.thrift.TApplicationException get_databases failed: out of sequence response)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:962)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabases(HiveMetaStoreClient.java:724)
    at org.apache.hive.service.cli.CLIService.start(CLIService.java:83)
    ... 3 more
2014-03-17 10:46:08,791 INFO  server.HiveServer2 (HiveStringUtils.java:run(622)) - SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down HiveServer2 at ASANDE1/16.155.82.203
************************************************************/
2014-03-17 10:46:22,805 INFO  server.HiveServer2 (HiveStringUtils.java:startupShutdownMessage(604)) - STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting HiveServer2
STARTUP_MSG:   host = ASANDE1/16.155.82.203
STARTUP_MSG:   args = [-hiveconf, hive.hadoop.classpath=c:\hdp\hive-0.12.0.2.0.6.0-0009\lib\*, -hiveconf, hive.querylog.location=c:\hadoop\logs\hive\history, -hiveconf, hive.log.dir=c:\hadoop\logs\hive]
STARTUP_MSG:   version = 0.12.0.2.0.6.0-0009
STARTUP_MSG:   classpath = c:\hdp\hadoop-2.2.0.2.0.6.0-0009\etc\hadoop;c:\hdp\hadoop-2.2.0.2.0.6.0-0009\share\hadoop\common\lib\activation-1.1.jar;c:\hdp\hadoop-2.2.0.2.0.6.0-
...............................................................
STARTUP_MSG:   build = git://sijenkins-vm3/cygdrive/d/w/bw/project/hive-monarch -r a7f54db5645b645500778b92e7fad8fab7738080; compiled by 'jenkins' on Fri Dec 20 18:29:58 PST 2013
************************************************************/
2014-03-17 10:46:23,677 INFO  service.CompositeService (SessionManager.java:init(60)) - HiveServer2: Async execution thread pool size: 100
2014-03-17 10:46:23,677 INFO  service.CompositeService (SessionManager.java:init(62)) - HiveServer2: Async execution wait queue size: 100
2014-03-17 10:46:23,678 INFO  service.CompositeService (SessionManager.java:init(64)) - HiveServer2: Async execution thread keepalive time: 10
2014-03-17 10:46:23,682 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited.
2014-03-17 10:46:23,682 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited.
2014-03-17 10:46:23,682 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited.
2014-03-17 10:46:23,683 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:ThriftBinaryCLIService is inited.
2014-03-17 10:46:23,683 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited.
2014-03-17 10:46:23,683 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started.
2014-03-17 10:46:23,683 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started.
2014-03-17 10:46:23,683 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started.
2014-03-17 10:46:23,694 INFO  hive.metastore (HiveMetaStoreClient.java:open(244)) - Trying to connect to metastore with URI thrift://ASANDE1:9083
2014-03-17 10:46:24,093 INFO  hive.metastore (HiveMetaStoreClient.java:open(322)) - Waiting 1 seconds before next connection attempt.
2014-03-17 10:46:25,093 INFO  hive.metastore (HiveMetaStoreClient.java:open(332)) - Connected to metastore.
2014-03-17 10:46:25,118 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:ThriftBinaryCLIService is started.
2014-03-17 10:46:25,122 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started.
2014-03-17 10:46:25,351 INFO  thrift.ThriftCLIService (ThriftBinaryCLIService.java:run(76)) - ThriftBinaryCLIService listening on 0.0.0.0/0.0.0.0:10001
2014-03-17 12:27:04,409 INFO  server.HiveServer2 (HiveStringUtils.java:startupShutdownMessage(604)) - STARTUP_MSG: 
/************************************************************</code>

Upvotes: 1

Views: 5118

Answers (2)

David Kjerrumgaard
David Kjerrumgaard

Reputation: 1076

The difference between the Select * from SomeTable; and Select colName from SomeTable; queries you ran is that the later, Select colName from SomeTable; requires information from the Hive Metastore service. Since you are looking for a specific column, hive must first check the hive schema definition which is stored in the Hive Metastore.

Your issue is due to the fact that you cannot connect to the Hive Metastore, as indicated by the following message in your log file:

org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out

Try restarting your Hive Metastore service.

Upvotes: 0

Neels
Neels

Reputation: 2543

The query that worked for you does not launch a Map Reduce Job. It looks like there is a problem while launching Map Reduce Jobs. Can you check the hive logs (default location, assuming you are running hiveserver2 as user hive) – /tmp/hive/hive.log to see if it has the full error message.

One common error that you might find in the logs is the permission denied error. Mostly the ODBC driver logs on to as user hadoop which has no write access thus causing a failure in starting a Map Reduce Job. If you change this user to hdfs, your problem might be solved.

Upvotes: 1

Related Questions