CODEWITHSUNDEEP
javahadoopapache-sparkgradleprotocol-buffers
Fisher Coder
Fisher Coder

Reputation: 3576

proto2 with Spark cannot run

I've a proto file with syntax proto2

Also, I need to use Spark (2.0.2) and HBase. My project is built using Gradle.

Right now, when I run my Java code, I get this error:

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 3, localhost): java.lang.NoSuchMethodError: com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List;
    at com.google.protobuf.GeneratedMessageV3$FieldAccessorTable.<init>(GeneratedMessageV3.java:1707)
    at com.google.protobuf.AnyProto.<clinit>(AnyProto.java:52)
    at com.google.protobuf.Any.getDescriptor(Any.java:129)
    at com.google.protobuf.util.JsonFormat$ParserImpl.buildWellKnownTypeParsers(JsonFormat.java:1100)
    at com.google.protobuf.util.JsonFormat$ParserImpl.<clinit>(JsonFormat.java:1094)
    at com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:277)

This stacktrace is very similar to the issue posted here, but the resolution posted there didn't apply for me.

What I tried:

I changed

compile group: 'com.google.protobuf', name: 'protobuf-java', version: '2.5.0'

to

compile group: 'org.spark-project.protobuf', name: 'protobuf-java', version: '2.4.1-shaded' as that link suggested, but still, the same error persisted.

Any help is greatly appreciated!

Upvotes: 2

Views: 1387

Answers (1)

Fisher Coder
Fisher Coder

Reputation: 3576

OK, here's the solution:

relocate is the way to rescue.

In my main program, I've a build.gradle like this:

dependencies {
    compile project(path: ':utils:hbasewrapper', configuration: 'shadow')
}
apply plugin: 'com.github.johnrengelman.shadow'

shadowJar {
    baseName = 'awesome-jar'
    zip64 true
    // This is to avoid the conflict between proto3 we are using and the proto2 hbase is using.
    relocate ('com.google.protobuf', 'importer.com.google.protobuf')
}

Then, I created another gradle package which has following build.gradle

group 'com.abc.hbasewrapper'
version '1.0-SNAPSHOT'

dependencies {
    compile group: 'org.apache.hbase', name: 'hbase-client', version: '1.3.0'
    compile group: 'org.apache.hbase', name: 'hbase-server', version: '1.2.2'
}

apply plugin: 'java'
apply plugin: 'com.github.johnrengelman.shadow'

shadowJar {
    baseName = 'hbasewrapper'
    version = null
    zip64 true
    relocate ('com.google.protobuf', 'hbasewrapper.com.google.protobuf')
    relocate ('io.netty', 'hbasewrapper.io.netty')
}

The root cause is due the HBase is referencing proto2 and my program is using proto3, thus causing this NoSuchMethodError error. The solution is to use relocate, to relocate the com.google.protobuf to a different gradle project which sole purpose is to build the hbase jar files that has proto2.

Hope it helps other fellow programmers! ^ ^

Upvotes: 2

Related Questions