How to convert protocol-buffer message to a HashMap in java?

Question

I have a protobuf message of the form

enum PolicyValidationType {
    Number = 0;
}


message NumberPolicyValidation {
    optional int64 maxValue = 1;
    optional int64 minValue = 2;
}

message PolicyObject {
    required string key = 1;
    optional string value = 2;
    optional string name = 3;
    optional PolicyValidationType validationType = 4;
    optional NumberPolicyValidation numberPolicyValidation = 5;
}

For example

policyObject {
      key: "sessionIdleTimeoutInSecs"
      value: "1800"
      name: "Session Idle Timeout"
      validationType: Number
      numberPolicyValidation {
        maxValue: 3600
        minValue: 5
      }
}

Can someone let me know how can I convert this to a Map like below:-

{validationType=Number, name=Session Idle Timeout, numberPolicyValidation={maxValue=3600.0, minValue=5.0}, value=1800, key=sessionIdleTimeoutInSecs}

One way I can think of is convert this to a json and then convert the json to map?

PolicyObject policyObject;
...
JsonFormat jsonFormat = new JsonFormat();
final String s = jsonFormat.printToString(policyObject);
Type objectMapType = new TypeToken<HashMap<String, Object>>() {}.getType();
Gson gson = new GsonBuilder().registerTypeAdapter(new TypeToken<HashMap<String,Object>>(){}.getType(), new PrimitiveDeserializer()).create();
Map<String, Object> mappedObject = gson.fromJson(s, objectMapType);

I think there must be some better way. Can someone suggest any better approach?

Answer 1

Be aware that both approaches described above (serialize/deserialize by tuk and custom converter by Zarnuk) will produce different outputs.

With the serialize/deserialize approach:

1. Field names in snake_case format will be automatically converted into camelCase. JsonFormat.printer() does this.
2. Numeric values will be converted to float. Gson does that for you.
3. Values of type Duration will be converted into strings with format durationInseconds + "s", i.e. "30s" for a duration of 30 seconds and "0.000500s" for a duration of 500,000 nanoseconds. JsonFormat.printer() does this.

With the custom converter approach:

1. Field names will remain as they are described on the proto file.
2. Integers and floats will keep their own type.
3. Values of type Duration will become objects with their corresponding fields.

To show the differences, here is a comparison of the outcomes of both approaches.

Original message (here is the proto file):

method_config {
  name {
    service: "helloworld.Greeter"
    method: "SayHello"
  }
  retry_policy {
    max_attempts: 5
    initial_backoff {
      nanos: 500000
    }
    max_backoff {
      seconds: 30
    }
    backoff_multiplier: 2.0
    retryable_status_codes: UNAVAILABLE
  }
}

With the serialize/deserialize approach:

{
   methodConfig=[ // field name was converted to cameCase
      {
         name=[
            {
               service=helloworld.Greeter,
               method=SayHello
            }
         ],
         retryPolicy={
            maxAttempts=5.0, // was integer originally
            initialBackoff=0.000500s, // was Duration originally
            maxBackoff=30s, // was Duration originally
            backoffMultiplier=2.0,
            retryableStatusCodes=[
               UNAVAILABLE
            ]
         }
      }
   ]
}

With the custom converter approach:

{
   method_config=[ // field names keep their snake_case format
      {
         name=[
            {
               service=helloworld.Greeter,
               method=SayHello
            }
         ],
         retry_policy={
            max_attempts=5, // Integers stay the same
            initial_backoff={ // Duration values remains an object
               nanos=500000
            },
            max_backoff={
               seconds=30
            },
            backoff_multiplier=2.0,
            retryable_status_codes=[
               UNAVAILABLE
            ]
         }
      }
   ]
}

Bottom line

So which approach is better?

Well, it depends on what you are trying to do with the Map<String, ?>. In my case, I was configuring a grpc client to be retriable, which is done via ManagedChannelBuilder.defaultServiceConfig API. The API accepts a Map<String, ?> with this format.

After several trials and errors, I figured that the defaultServiceConfig API assumes you are using GSON, hence the serialize/deserialize approach worked for me.

One more advantage of the serialize/deserialize approach is that the Map<String, ?> can be easily converted back to the original protobuf value by serializing it back to json, then using the JsonFormat.parser() to obtain the protobuf object:

    ServiceConfig original;
    ...
    String asJson = JsonFormat.printer().print(original);
    Map<String, ?> asMap = new Gson().fromJson(asJson, Map.class);
    // Convert back to ServiceConfig
    String backToJson = new Gson().toJson(asMap);
    ServiceConfig.Builder builder = ServiceConfig.newBuilder();
    JsonFormat.parser().merge(backToJson, builder);
    ServiceConfig backToOriginal = builder.build();

... whereas the custom converter approach method doesn't have an easy way to convert back as you need to write a function to convert the map back to the original proto by navigating the tree.

Answer 2

I created small dedicated class to generically convert any Google protocol buffer message into a Java Map.

public class ProtoUtil {

@NotNull
public Map<String, Object> protoToMap(Message proto) {
    final Map<Descriptors.FieldDescriptor, Object> allFields = proto.getAllFields();
    Map<String, Object> map = new LinkedHashMap<>();
    for (Map.Entry<Descriptors.FieldDescriptor, Object> entry : allFields.entrySet()) {
        final Descriptors.FieldDescriptor fieldDescriptor = entry.getKey();
        final Object requestVal = entry.getValue();
        final Object mapVal = convertVal(proto, fieldDescriptor, requestVal);
        if (mapVal != null) {
            final String fieldName = fieldDescriptor.getName();
            map.put(fieldName, mapVal);
        }
    }
    return map;
}


@Nullable
/*package*/ Object convertVal(@NotNull Message proto, @NotNull Descriptors.FieldDescriptor fieldDescriptor, @Nullable Object protoVal) {
    Object result = null;
    if (protoVal != null) {
        if (fieldDescriptor.isRepeated()) {
            if (proto.getRepeatedFieldCount(fieldDescriptor) > 0) {
                final List originals = (List) protoVal;
                final List copies = new ArrayList(originals.size());
                for (Object original : originals) {
                    copies.add(convertAtomicVal(fieldDescriptor, original));
                }
                result = copies;
            }
        } else {
            result = convertAtomicVal(fieldDescriptor, protoVal);
        }
    }
    return result;
}


@Nullable
/*package*/ Object convertAtomicVal(@NotNull Descriptors.FieldDescriptor fieldDescriptor, @Nullable Object protoVal) {
    Object result = null;
    if (protoVal != null) {
        switch (fieldDescriptor.getJavaType()) {
            case INT:
            case LONG:
            case FLOAT:
            case DOUBLE:
            case BOOLEAN:
            case STRING:
                result = protoVal;
                break;
            case BYTE_STRING:
            case ENUM:
                result = protoVal.toString();
                break;
            case MESSAGE:
                result = protoToMap((Message) protoVal);
                break;
        }
    }
    return result;
}


}

Hope that helps! Share and enjoy.