CODEWITHSUNDEEP
javaapache-sparkprotocol-buffers
Andrei Tsukanov
Andrei Tsukanov

Reputation: 53

spark-submit protobuf openrtb java.lang.VerifyError

I have gzipped files on s3, files contain encoded Base64 string representing byte array of the protobuf message. Proto scheme looks like:

syntax = "proto2";
package com.myproject.proto;
option java_outer_classname = "MyProtos";
import "openrtb.proto";

message Request {
    optional int64 timestamp = 1;
    optional com.google.openrtb.BidRequest bidRequest = 2;
    optional string otherData = 3;
}

When I running next spark code for flatMap function from local:

public static Iterator<MyProtos.Request> parseRequest(String source) {
    try {
        byte[] bytes = Base64.decodeBase64(source);
        MyProtos.Request request = MyProtos.Request.parseFrom(bytes);
        return Collections.singletonList(request).iterator();
    } catch (Exception e) {
        return Collections.emptyIterator();
    }
}

everything is ok, but when I try run this code on remote by spark-submit I got exception:

java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    com/google/protobuf/GeneratedMessageV3$ExtendableMessage.hasExtension(Lcom/google/protobuf/GeneratedMessage$GeneratedExtension;)Z @2: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @2
    flags: { }
    locals: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
    stack: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0x0000000: 2a2b b600 21ac

Upvotes: 2

Views: 1002

Answers (2)

webber
webber

Reputation: 1886

The issue in my case was that the app was built with protobuf 3.5.0 but Spark had 2.5.0 in the jars dir. Simple resolution was to drop the new 3.5.0 jar in.

Upvotes: 1

Andrei Tsukanov
Andrei Tsukanov

Reputation: 53

The problem was in Runtime Enviroment variable spark.executor.userClassPathFirst by default equals false. If run spark on remote or locally in client mode, there was no such dependencies conflict problem.

Upvotes: 0

Related Questions