Reputation: 3
After configuring Hive with Tez, it works properly and the ACID table is created successfully. but when trying to run insert statement to insert record to this table, Tez failed withe below log
2023-08-04 09:48:26,265 [INFO] [main] |common.TezClassLoader|: Created TezClassLoader with parent classloader: sun.misc.Launcher$AppClassLoader@677327b6, thread: 1, system classloader: sun.misc.Launcher$AppClassLoader@677327b6
2023-08-04 09:48:27,355 [INFO] [main] |app.DAGAppMaster|: Creating DAGAppMaster for applicationId=application_1691002829482_0026, attemptNum=1, AMContainerId=container_1691002829482_0026_01_000001, jvmPid=83708, userFromEnv=hadoop, cliSessionOption=true, pwd=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1691002829482_0026/container_1691002829482_0026_01_000001, localDirs=/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1691002829482_0026, logDirs=/opt/hadoop-3.1.1/logs/userlogs/application_1691002829482_0026/container_1691002829482_0026_01_000001
2023-08-04 09:48:28,487 [INFO] [main] |security.SecurityUtil|: Updating Configuration
2023-08-04 09:48:29,360 [INFO] [main] |app.DAGAppMaster|: Created DAGAppMaster for application appattempt_1691002829482_0026_000001, versionInfo=[ component=tez-dag, version=0.10.2, revision=72977b8720b2337ab0a0a3bf3b12e1c57900fa69, SCM-URL=scm:git:https://gitbox.apache.org/repos/asf/tez.git, buildTime=2022-07-08T16:21:10Z, buildUser=laszlobodor, buildJavaVersion=1.8.0_292 ]
2023-08-04 09:48:29,624 [INFO] [main] |app.DAGAppMaster|:
/************************************************************
[system properties]
os.name: Linux
os.version: 3.10.0-1160.el7.x86_64
java.home: /opt/jdk1.8.0_202/jre
java.runtime.version: 1.8.0_202-b08
java.vendor: Oracle Corporation
java.version: 1.8.0_202
java.vm.name: Java HotSpot(TM) 64-Bit Server VM
java.class.path: /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1691002829482_0026/container_1691002829482_0026_01_000001:/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1691002829482_0026/container_1691002829482_0026_01_000001/hive-exec-3.1.2-50b0545693ae9392559584116a172ee32922e57ae9ea0eb7dc120290c217c2d5.jar:/tmp/hadoop-hadoop/nm-local-
user.name: hadoop
************************************************************/
2023-08-04 09:48:29,717 [INFO] [main] |counters.Limits|: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2023-08-04 09:48:29,717 [INFO] [main] |counters.Limits|: Counter limits initialized with parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=120
2023-08-04 09:48:29,751 [INFO] [main] |shim.HadoopShimsLoader|: Trying to locate HadoopShimProvider for hadoopVersion=3.3.1, majorVersion=3, minorVersion=3
2023-08-04 09:48:29,754 [INFO] [main] |shim.HadoopShimsLoader|: Picked HadoopShim org.apache.tez.hadoop.shim.DefaultHadoopShim, providerName=null, overrideProviderViaConfig=null, hadoopVersion=3.3.1, majorVersion=3, minorVersion=3
2023-08-04 09:48:30,056 [INFO] [main] |app.DAGAppMaster|: AM Level configured TaskSchedulers: [0:TezYarn:null],[1:TezUber:null]
2023-08-04 09:48:30,056 [INFO] [main] |app.DAGAppMaster|: AM Level configured ContainerLaunchers: [0:TezYarn:null],[1:TezUber:null]
2023-08-04 09:48:30,056 [INFO] [main] |app.DAGAppMaster|: AM Level configured TaskCommunicators: [0:TezYarn:null],[1:TezUber:null]
2023-08-04 09:48:30,056 [INFO] [main] |app.DAGAppMaster|: Comparing client version with AM version, clientVersion=0.10.2, AMVersion=0.10.2
2023-08-04 09:48:31,885 [INFO] [main] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster failed in state INITED
java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to com.google.protobuf.Message
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:123)
at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:965)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1731)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1752)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
at org.apache.tez.common.TezCommonUtils.getTezBaseStagingPath(TezCommonUtils.java:88)
at org.apache.tez.common.TezCommonUtils.getTezSystemStagingPath(TezCommonUtils.java:146)
at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:484)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2600)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2597)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2384)
2023-08-04 09:48:31,893 [WARN] [main] |service.AbstractService|: When stopping the service org.apache.tez.dag.app.DAGAppMaster
java.lang.NullPointerException
at org.apache.tez.dag.app.DAGAppMaster.initiateStop(DAGAppMaster.java:2111)
at org.apache.tez.dag.app.DAGAppMaster.serviceStop(DAGAppMaster.java:2126)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
at org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
at org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2600)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2597)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2384)
2023-08-04 09:48:31,893 [ERROR] [main] |app.DAGAppMaster|: Error starting DAGAppMaster
java.lang.ClassCastException: org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$GetFileInfoRequestProto cannot be cast to com.google.protobuf.Message
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:123)
at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:965)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1731)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1752)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1749)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1764)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1760)
at org.apache.tez.common.TezCommonUtils.getTezBaseStagingPath(TezCommonUtils.java:88)
at org.apache.tez.common.TezCommonUtils.getTezSystemStagingPath(TezCommonUtils.java:146)
at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:484)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2600)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2597)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2384)
2023-08-04 09:48:31,941 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked
Service org.apache.tez.dag.app.DAGAppMaster failed in state INITED i tried some solutions with no luck, could you please help me in that
Upvotes: 0
Views: 139