CODEWITHSUNDEEP
pythonprotocol-buffersprotobuf-python
Nuno Mariz
Nuno Mariz

Reputation: 589

Protobuf 3.19.1 and Python 3.10 issue

This is all about and issue when using the latest Python Protobuf (3.19.1) and Python 3.10, in Linux (tested in Fedora 35 and Ubuntu 20.04.

It broke our library but it can easily tested using the addressbook.proto from the Python Protobuf tutorial and tried to get the proto2 message class as follows:

import addressbook_pb2

from google.protobuf import (
    descriptor_database,
    descriptor_pb2,
    descriptor_pool,
    message_factory,
)

_DESCRIPTOR_DB = descriptor_database.DescriptorDatabase()
_DESCRIPTOR_POOL = descriptor_pool.DescriptorPool(_DESCRIPTOR_DB)

_DESCRIPTOR_DB.Add(
    descriptor_pb2.FileDescriptorProto.FromString(
        addressbook_pb2.DESCRIPTOR.serialized_pb
    )
)
factory = message_factory.MessageFactory()
cls = factory.GetPrototype(_DESCRIPTOR_POOL.FindMessageTypeByName("tutorial.Person"))

It raises the following error:

[libprotobuf ERROR google/protobuf/pyext/descriptor_database.cc:64] DescriptorDatabase method raised an error
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
Traceback (most recent call last):
  File "/dev/protobuf/test/test.py", line 21, in <module>
    ls = factory.GetPrototype(_DESCRIPTOR_POOL.FindMessageTypeByName("tutorial.Person"))
`KeyError: "Couldn't find message tutorial.Person"

Now, it works as expected if I use an older Python Protobuf version, such as 3.18.1.

I've opened a bug https://github.com/protocolbuffers/protobuf/issues/9245, but apparently, it was not considered a bug.

Python Protobuf introduced the PY_SSIZE_T_CLEAN macro in 3.19.1 and broke something, probably by using int instead of Py_ssize_t when using # formats.

Have anyone have this issue or can confirm it?

Upvotes: 1

Views: 5594

Answers (2)

jpa
jpa

Reputation: 12176

There are actually two errors here:

SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

This error is caused by Python 3.10 dropping support for old default conversions when passing data from C to Python side. In this case in the protobuf library, the error only occurs when passing an exception from C code to Python.

The python-protobuf library was fixed to work with Python 3.10 back in October 2021, and the fix should be included in python-protobuf 3.20.0 and later.

Try adding this to your script to check the version:

import google.protobuf
print(google.protobuf.__version__)

For me the error does not occur with the latest versions 3.19.4, 3.20.1 or 4.21.1, but does occur with 3.19.2 and older.

Upvotes: 2

Divya Nambiar
Divya Nambiar

Reputation: 149

Yes I am also getting same issue.

We changes the version as below:

*protobuf >= 3.19* # Not working

*protobuf >= 3.15.6, <= 3.20.1* # Working

Upvotes: 1

Related Questions