Stack trace: ExitCodeException exitCode=1 when starting MapReduce job on Bigtable

Question

We are using Google Cloud Bigtable for our Big Data. When I'm running MapReduce job I assembly a jar and run it and now I'm getting this error:

Application application_1451577928704_0050 failed 2 times due to AM Container for appattempt_1451577928704_0050_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://censored:8088/cluster/app/application_1451577928704_0050Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e02_1451577928704_0050_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 Failing this attempt. Failing the application.

When I logged to see the logging of the workers node I saw this error:

2016-02-15 02:59:54,106 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1451577928704_0050_000001 2016-02-15 02:59:54,294 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2016-02-15 02:59:54,319 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2016-02-15 02:59:54,319 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 50 cluster_timestamp: 1451577928704 } attemptId: 1 } keyId: -******) 2016-02-15 02:59:54,424 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. 2016-02-15 02:59:54,755 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2016-02-15 02:59:54,855 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2016-02-15 02:59:54,911 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ClassCastException: org.apache.xerces.dom.DeferredElementNSImpl cannot be cast to org.w3c.dom.Text org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.ClassCastException: org.apache.xerces.dom.DeferredElementNSImpl cannot be cast to org.w3c.dom.Text at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:478) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:458) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1560) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:458) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:377) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448) Caused by: java.lang.ClassCastException: org.apache.xerces.dom.DeferredElementNSImpl cannot be cast to org.w3c.dom.Text at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2603) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2502) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2405) at org.apache.hadoop.conf.Configuration.get(Configuration.java:981) at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1031) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1432) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:105) at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:184) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:474) ... 11 more

---

I tried an older jar and it's running perfectly fine and I'm not sure why the new jar won't work - Didn't change anything.

Please advise?

Thanks!

Update 1: Here is some more details: I setup the cluster with the dataproc. We are using the newest versions, here is the library dependencies:

val BigtableHbase = "com.google.cloud.bigtable" % "bigtable-hbase-1.1" % "0.2.2" val BigtableHbaseMapreduce = "com.google.cloud.bigtable" % "bigtable-hbase-mapreduce" % "0.2.2" val CommonsCli = "commons-cli" % "commons-cli" % "1.2" val HadoopCommon = "org.apache.hadoop" % "hadoop-common" % "2.7.1" val HadoopMapreduceClientApp = "org.apache.hadoop" % "hadoop-mapreduce-client-app" % "2.7.1" val HbaseCommon = "org.apache.hbase" % "hbase-common" % "1.1.2" val HbaseProtocol = "org.apache.hbase" % "hbase-protocol" % "1.1.2" val HbaseClient = "org.apache.hbase" % "hbase-client" % "1.1.2" val HbaseServer = "org.apache.hbase" % "hbase-server" % "1.1.2" val HbaseAnnotations = "org.apache.hbase" % "hbase-annotations" % "1.1.2"

libraryDependencies += BigtableHbase libraryDependencies += BigtableHbaseMapreduce libraryDependencies += CommonsCli libraryDependencies += HadoopCommon libraryDependencies += HadoopMapreduceClientApp libraryDependencies += HbaseCommon libraryDependencies += HbaseProtocol libraryDependencies += HbaseClient libraryDependencies += HbaseServer libraryDependencies += HbaseAnnotations

Java version:

openjdk version "1.8.0_66-internal" OpenJDK Runtime Environment (build 1.8.0_66-internal-b17) OpenJDK 64-Bit Server VM (build 25.66-b17, mixed mode)

Alpn version: alpn-boot-8.1.3.v20150130

hbase verison:

2016-02-15 20:45:42,050 INFO [main] util.VersionInfo: HBase 1.1.2 2016-02-15 20:45:42,051 INFO [main] util.VersionInfo: Source code repository file:///mnt/ram/bigtop/bigtop/output/ hbase/hbase-1.1.2 revision=Unknown 2016-02-15 20:45:42,051 INFO [main] util.VersionInfo: Compiled by bigtop on Tue Nov 10 19:09:17 UTC 2015 2016-02-15 20:45:42,051 INFO [main] util.VersionInfo: From source with checksum 42e8a1890c700d37485c69a44a3

hadoop version:

Hadoop 2.7.1 Subversion https://bigdataoss-internal.googlesource.com/third_party/apache/bigtop -r 2a194d4d838b79460c3ceb892f3c94 44218ba970 Compiled by bigtop on 2015-11-10T18:38Z Compiled with protoc 2.5.0 From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a This command was run using /usr/lib/hadoop/hadoop-common-2.7.1.jar

Answer 1

I found the problem in my case! The hbase-site.xml was slightly different in the hbase.client.connection.impl property.

<property>
    <name>hbase.client.connection.impl</name>
    <value>com.google.cloud.bigtable.hbase1_1.BigtableConnection</value>
</property>

I got to this after extracting and comparing the two jars.

Answer 2

The newer versions of the bigtable client jar include newer versions of the gRPC jar. Newer versions of the gRPC jar depend on newer versions of alpn-boot or OpenSSL. In addition to a new version of the bigtable jar, you may need a new version of the alpn-boot jar. Unfortunately, the Jetty team isn't making new alpn-boot jars for Java7, which bdutil depends on.

We are actively working on moving away from bdutil to dataproc, which is the newer version of Google Cloud Hadoop management. Dataproc uses Java 8, and doesn't have the same problems as bdutil. There are still kinks we need to work out.

More information can be found at:

https://cloud.google.com/dataproc/examples/cloud-bigtable-example and https://github.com/grpc/grpc-java/blob/master/SECURITY.md