CODEWITHSUNDEEP
pythonjsonprotocol-buffers
exHash
exHash

Reputation: 1265

Protobuf to json in python

I have an object that I de-serialize using protobuf in Python. When I print the object it looks like a python object, however when I try to convert it to json I have all sorts of problems.

For example, if I use json.dumps() I get that the object (the generated code from protoc) does not contain a _ dict _ error.

If I use jsonpickle I get UnicodeDecodeError: 'utf8' codec can't decode byte 0x9d in position 97: invalid start byte.

Test code below is using jsonpickle with the error shown above. 

if len(sys.argv) < 2:
    print ("Error: missing ser file")
    sys.exit()
else :
    fileLocation = sys.argv[1]

org = BuildOrgObject(fileLocation) 

org = org.Deserialize()


#print (org)
jsonObj = jsonpickle.encode(org)
print (jsonObj)

Upvotes: 114

Views: 166482

Answers (6)

jm-nab
jm-nab

Reputation: 94

In my case I was trying to convert google vision api response protobuf to dict.

You can try:

from google.protobuf.json_format import MessageToDict


(Pdb) type(response)
<class 'google.cloud.vision_v1.types.image_annotator.BatchAnnotateFilesResponse'>

print(MessageToDict(response._pb))
{'responses': [{'responses': [...]},...]}

If you try MessageToDict with just the response you'll get a missing DESCRIPTOR error

Upvotes: 1

Nirav
Nirav

Reputation: 81

Here's my function to convert a proto3 object to a JSON object (i.e. Python dictionary):

def protobuf_to_dict(proto_obj):
    key_list = proto_obj.DESCRIPTOR.fields_by_name.keys()
    d = {}
    for key in key_list:
        d[key] = getattr(proto_obj, key)
    return d

Since the converters from Google's protobuf library don't seem to work in some cases with the 3.19 version, this function leverages the Descriptor class present on each Protobuf object.

Here, getattr(obj, string_attribute) returns the value given by obj.attribute

Upvotes: 3

vr286
vr286

Reputation: 836

If you are using an older version that doesn't has the preserving_proto_field_name field:

from google.protobuf.json_format import MessageToJson
def proto_to_json(proto_obj):
    json_obj = MessageToJson(proto_obj):
    json_obj = MessageToJso, including_default_value_fields=True)
    # Change lowerCamelCase of google Json conversion to the snake_case as in original protobuf
    dict_obj = dict((re.sub(r'(?<!^)(?=[A-Z])', '_', k).lower(),v) for k, v in json.loads(json_obj).items())
    if hasattr(proto_obj, 'uuid'):
        dict_obj["uuid"] = proto_obj.uuid.encode("hex")
    return json.dumps(dict_obj, indent=4, sort_keys=True)

Upvotes: 0

denis-sumin
denis-sumin

Reputation: 2611

I'd recommend using protobuf↔json converters from google's protobuf library:

from google.protobuf.json_format import MessageToJson

json_obj = MessageToJson(org)

You can also serialise the protobuf to a dictionary:

from google.protobuf.json_format import MessageToDict
dict_obj = MessageToDict(org)

Refer to the protobuf package API documentation: https://developers.google.com/protocol-buffers/docs/reference/python/ (see module google.protobuf.json_format).

Upvotes: 249

Shubham Kanyal
Shubham Kanyal

Reputation: 37

You can also user SerializeToString.

org.SerializeToString()

Upvotes: 2

Kevin Hill
Kevin Hill

Reputation: 493

If you need to go straight to json take a look at the protobuf-to-json library, but you'll have to install that manually.

But I would recommend that you use the protobuf-to-dict library instead for a few reasons:

  1. It is accessible from pypi so you can simply pip install protobuf-to-dict or include it in a requirements.txt
  2. dict can be converted to json and might be more useful than a json string

Upvotes: 17

Related Questions