CODEWITHSUNDEEP
javahive
CK Chang
CK Chang

Reputation: 33

[Hive]I got "ArrayIndexOutOfBoundsException" while I query the hive database

I always get "ArrayIndexOutOfBoundsException" while I query the hive base(both hive-0.11.0 and hive-0.12.0), but sometimes not. Here is the error

java.lang.RuntimeException: Hive Runtime Error while closing operators: java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:313)
at org.apache.hadoop.io.IOUtils.cleanup(IOUtils.java:232)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:539)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException: 0
    at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:231)
    at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
    at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:645)
    at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:758)
    at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:257)
    at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:298)
    ... 8 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
    at org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:220)
    ... 13 more

Could someone help me?

Update my code:

    Select distinct jabberUseragent.gameID,agentPlayInfo.gameLabel,jabberUseragent.userAgent,CONCAT(CONCAT(CONCAT(triggerUsageStart.generateDate,' '),triggerUsageStart.timezone),CONCAT(' ',triggerUsageStart.generateTime)) as generateDate,(unix_timestamp(CONCAT(CONCAT(triggerUsageStop.generateDate,' '),triggerUsageStop.generateTime)) - unix_timestamp(CONCAT(CONCAT(triggerUsageStart.generateDate,' '),triggerUsageStart.generateTime))) from
    (Select gameSession,gameID,userAgent from(Select distinct regexp_extract(t.payload,'playRequestId:(.*), playRequest',1) as gameSession,regexp_extract(t.payload,'gameId:(.*), userAgent:',1) as gameID,regexp_extract(t.payload,', userAgent:(.*), agentLocation',1) as userAgent,payload from (select * from ${hiveconf:DATA_BASE} p where p.dt >= '${hiveconf:LOW_DATE}' and p.dt <= '${hiveconf:UPPER_DATE}')  t where CONCAT(t.generatedate,t.generatetime) >= CONCAT('${hiveconf:LOW_DATE}','${hiveconf:LOW_TIME}') and CONCAT(t.generatedate,t.generatetime) <= CONCAT('${hiveconf:UPPER_DATE}','${hiveconf:UPPER_TIME}'))jabberUseragent where jabberUseragent.gameSession!='' and jabberUseragent.userAgent!='') jabberUseragent
join
    (Select gameID,gameLabel from(Select distinct regexp_extract(t.payload,'gameId=(.*),gameLabel=.*,configFilePath',1) as gameID,regexp_extract(t.payload,'gameId=.*,gameLabel=(.*),configFilePath',1) as gameLabel,payload from (select * from ${hiveconf:DATA_BASE} p where p.dt >= '${hiveconf:LOW_DATE}' and p.dt <= '${hiveconf:UPPER_DATE}')  t where CONCAT(t.generatedate,t.generatetime) >= CONCAT('${hiveconf:LOW_DATE}','${hiveconf:LOW_TIME}') and CONCAT(t.generatedate,t.generatetime) <= CONCAT('${hiveconf:UPPER_DATE}','${hiveconf:UPPER_TIME}'))agentPlayInfo where agentPlayInfo.gameID!='' and agentPlayInfo.gameLabel!='') agentPlayInfo
join
    (Select gameSession,generateDate,generateTime,timezone,payload from(Select distinct regexp_extract(t.payload,'GAME_SESSION=.*((.{8})-(.{4})-(.{4})-(.{4})-(.{12}))\" USAGE=\"([\\w\\-\\(\\)\\.]*,){41}9.*\"',1) as gameSession,generateDate,generateTime,timezone,payload from (select * from ${hiveconf:DATA_BASE} p where p.dt >= '${hiveconf:LOW_DATE}' and p.dt <= '${hiveconf:UPPER_DATE}')  t where t.payload like '%[e] usage_record%' and CONCAT(t.generatedate,t.generatetime) <= CONCAT('${hiveconf:UPPER_DATE}','${hiveconf:UPPER_TIME}') and CONCAT(t.generatedate,t.generatetime) >= CONCAT('${hiveconf:LOW_DATE}','${hiveconf:LOW_TIME}'))triggerStart where triggerStart.gameSession!='')triggerUsageStart
join
    (Select gameSession,generateDate,generateTime,timezone,payload from(Select distinct regexp_extract(t.payload,'GAME_SESSION=.*((.{8})-(.{4})-(.{4})-(.{4})-(.{12}))\" USAGE=\"([\\w\\-\\(\\)\\.]*,){41}[1-5].*\"',1) as gameSession,generateDate,generateTime,timezone,payload from (select * from ${hiveconf:DATA_BASE} p where p.dt >= '${hiveconf:LOW_DATE}' and p.dt <= '${hiveconf:UPPER_DATE}')  t where t.payload like '%[e] usage_record%' and CONCAT(t.generatedate,t.generatetime) <= CONCAT('${hiveconf:UPPER_DATE}','${hiveconf:UPPER_TIME}') and CONCAT(t.generatedate,t.generatetime) >= CONCAT('${hiveconf:LOW_DATE}','${hiveconf:LOW_TIME}'))triggerStop where triggerStop.gameSession!='')triggerUsageStop
on (jabberUseragent.gameSession = triggerUsageStart.gameSession and triggerUsageStart.gameSession = triggerUsageStop.gameSession and jabberUseragent.gameID = agentPlayInfo.gameID) order by generateDate

Sorry, I can't share my samples. By the way, I've also got this exception before I got "ArrayIndexOutOfBoundException"

javax.jdo.JDODataStoreException: Error executing SQL query "select PARTITIONS.PART_ID from PARTITIONS  inner join TBLS on PARTITIONS.TBL_ID = TBLS.TBL_ID   inner join DBS on TBLS.DB_ID = DBS.DB_ID  where TBLS.TBL_NAME = ? and DBS.NAME = ? and PARTITIONS.PART_NAME in (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)".
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOQuery.executeWithArray(JDOQuery.java:321)
at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilterInternal(MetaStoreDirectSql.java:181)
at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getPartitionsViaSqlFilter(MetaStoreDirectSql.java:82)
at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNamesInternal(ObjectStore.java:1717)
at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByNames(ObjectStore.java:1700)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)......

NestedThrowablesStackTrace:
org.postgresql.util.PSQLException: ERROR: relation "partitions" does not exist
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:1591)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1340)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:192)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:471)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:373)
    at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:258)......

Upvotes: 2

Views: 9899

Answers (1)

Keerthivasan
Keerthivasan

Reputation: 12890

Based on the information provided, only this can be a sensible solution to your problem.

I have put the method definition for reference. Please go through it to understand

If you run down the source code closely, there are two areas / possibilities where ArrayIndexOutOfBoundException can be thrown.

  • Accessing the array values of input splits from Configuration
  • Reading the Row from currentReadBlock array (this is mostly not the case for the exception since it's size is greater than 0)

Please check your set of input files for job because InputFormat#split() method returns an array of InputSplit type. Each InputSplit is then assigned to an individual Mapper for processing. Mostly, the exception occurs while accessing this InputSplit[] array.

Upvotes: 1

Related Questions