CODEWITHSUNDEEP
pythonpython-3.xthrift
Rtik88
Rtik88

Reputation: 1787

Apache Thrift Python 3 support

I compiled my test.thrift file using:

thrift -gen py test.thrift

Then i tried to import the created files:

from test.ttypes import *

When I use Python 2.7 the import works but with Python 3.4 it raises

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/art/SerTest/addressThrift/gen-py/test/ttypes.py", line11, in <module>
from thrift.transport import TTransport
File "/usr/local/lib/python3.4/dist-
packages/thrift/transport/TTransport.py", line 20, in <module>  
from cStringIO import StringIO
ImportError: No module named 'cStringIO'

I tried to run: sudo python3 setup.py install and got many exceptions, all seems to be related to python 2 vs 3 problems. for example:

File "/usr/local/lib/python3.4/dist-
     packages/thrift/transport/TSSLSocket.py", line 99
except socket.error, e:
                   ^
SyntaxError: invalid syntax

I addition there is a warning thats seems important:

/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'use_2to3'

Googling Thrift Python 3 support seems contradicting.
Those say that there is no support:
Does cql support python 3?
https://issues.apache.org/jira/browse/THRIFT-1857
And here I understand from the subtext that it does:
Thrift python 3.4 TypeError: string argument expected, got 'bytes'
Python 3.4 TypeError: input expected at most 1 arguments, got 3
https://issues.apache.org/jira/browse/THRIFT-2096
So does Thrift support Python 3.X? If so what did I missed?

Upvotes: 8

Views: 11654

Answers (3)

ygormutti
ygormutti

Reputation: 357

It seems that thrift 0.10.0, released in January 2017, already has built-in support for Python 2 and 3 at the same time, but beware there are some compatibility issues.

Sources:

Upvotes: 3

Adirio
Adirio

Reputation: 5286

Install thrift with pip3 to make sure it gets converted (I think they use 2to3) and it will use the python3 syntax except ExceptionName as e as well as other python3 syntax like turning print into a function, ...

However, there are still some bugs. I found that pack() results are used as input to a StringIO as in python2 it returns a str but in python3 it was changed so that it returns a bytes. The easiest solution is to change the following files:

transport/TTransport.py:

Line 20:

from io import StringIO

to (option 1)

from io import BytesIO as StringIO

or (option 2)

from io import BytesIO

Line 55:

buff = ''

to

buff = b''

Line 142:

self.__rbuf = StringIO("")

to

self.__rbuf = StringIO(b'')

Aditional changes for option 2:

Search every

StringIO

and replace it for

BytesIO

protocol/TBinaryProtocol.py

Lines 122-123

self.writeI32(len(str))
self.trans.write(str)

to

buff = bytes(str, 'utf-8')
self.writeI32(len(buff))
self.trans.write(buff)

Line 223

str = self.trans.readAll(len)

to

str = self.trans.readAll(len).decode('utf-8')

I also made 2 functions that are equivalent to python2's pack() and unpack() for python3 that may be useful for some other problems where you can't just substitute the io for the proper one:

from io import pack, unpack

def packStr(fmt, v):
  return "".join(["\\x{:02x}".format(i) for i in list(pack(fmt, v))])

def unpackStr(fmt, v):
  return unpack(fmt, bytes([int(i, 16) for i in v.split("\\x")[1:]]))

Upvotes: 4

WooParadog
WooParadog

Reputation: 649

There is a better solution to this. Instead of waiting the official python 3 support, why not use our python implementation of thrift?

https://github.com/eleme/thriftpy

it fully supports python 3, and pypy, and pypy3.

Upvotes: 7

Related Questions