CODEWITHSUNDEEP
pythoncpython-extensions
rivenmyst137
rivenmyst137

Reputation: 365

Benefit of converting Python method to C extension?

A relatively simple question:
If I convert a CPU-bound bottleneck method from Python to a C extension (roughly implementing the same algorithm),

UPDATE: People seemed to be complaining on the lack of specifics. I was mostly trying to understand what factors would make a piece of Python code a good candidate for being rewritten in C (i.e., when would porting to C actually give you a speed boost if the original Python is CPU-bound).

For specifics, this is the piece of code I'm looking at. Basically it's a recursive method that takes two lists of lists (a list of "columns", where each column contains possible values that could go in that column...basically, a schema), and seeing if it's possible to make less than n (usually 1) change(s) (where a change might be to add a new value to a column, add a new column, remove a column, etc.) such that there's some sequence of values (one value from each column) you could construct out of either schema. It's very similar in spirit to calculating the edit distance between to strings. Here's the code:

def CheckMerge(self, schemai, schemaj, starti, startj, \
               changesLeft, path):
#        if starti == 0 and startj == 0:
#            print '\n'
#            print schemai.schema
#            print ''
#            print schemaj.schema
    if starti == len(schemai.schema) and startj == len(schemaj.schema):
        return (True, path)
    if starti < len(schemai.schema):
        icopy = schemai.schema[starti]
    else:
        icopy = []
    if startj < len(schemaj.schema):
        jcopy = schemaj.schema[startj]
    else:
        jcopy = []
    intersect = set(icopy).intersection(set(jcopy))
    intersect.discard('')
    if len(intersect) == 0:
        if starti < len(schemai.schema) and \
            ('' in schemai.schema[starti] or changesLeft > 0):

            if not '' in schemai.schema[starti]:
                changesLeft -= 1
            changesCopy = list(path)
            changesCopy.append('skipi')
            result,steps = self.CheckMerge(schemai, schemaj, starti+1, startj, \
                                     changesLeft, changesCopy)
            if result:
                return (result,steps)
            elif not '' in schemai.schema[starti]:
                changesLeft += 1

        if startj < len(schemaj.schema) and \
            ('' in schemaj.schema[startj] or changesLeft > 0):

            if not '' in schemaj.schema[startj]:
                changesLeft -= 1
            changesCopy = list(path)
            changesCopy.append('skipj')
            result,steps = self.CheckMerge(schemai, schemaj, starti, startj+1, \
                                     changesLeft, changesCopy)
            if result:
                return (result, steps)
            elif not '' in schemaj.schema[startj]:
                changesLeft += 1

        if changesLeft > 0:
            changesCopy = list(path)
            changesCopy.append('replace')
            changesLeft -= 1
            result,steps = self.CheckMerge(schemai, schemaj, starti+1, startj+1, \
                                     changesLeft, changesCopy)
            if result:
                return (result, steps)

        return (False, None)
    else:
        changesCopy = list(path)
        changesCopy.append('merge')
        result,steps = self.CheckMerge(schemai, schemaj, starti+1, startj+1, \
                                     changesLeft, changesCopy)
        if result:
            return (result, steps)
        else:
            return (False, None)

Upvotes: 1

Views: 172

Answers (2)

octopusgrabbus
octopusgrabbus

Reputation: 10695

Python runs pretty fast, so you would need a distinct reason to convert a Python function to C, like to access hardware, which has already been mentioned. But, here is another reason.

Python (C Python) suffers from the Global Interpreter Lock (GIC) problem. Python threads cannot run simultaneously, only one at a time. So, you could put thread-specific code into C, which is not restricted by the GIC problem.

In general, if you believe your Python code is slow and it there is not a specific reason as you have mentioned in your post, then you may need to adapt to more Python-ic coding conventions, like list comprehensions and other features found in Python and not too many other languages.

My final comment is not a reflection on your code sample. Instead I am supplying it as the general wisdom that I've learned listening to a lot of Python presentations.

Upvotes: 0

John Smith
John Smith

Reputation: 1048

That solely and completely depends on your code. If some piece of your code is supported by the hardware, like, if you're computing the Hamming weight, doing AES encrption, calculating CRC, or have a vectorizable code, there are hardware instructions for them that boosts up the speed, and you can accesss them by C code but not python code.

Upvotes: 1

Related Questions