CODEWITHSUNDEEP
pythoncnumpy
Mickey Diamant
Mickey Diamant

Reputation: 647

best way to extend python / numpy performancewise

As there are multitude of ways to write binary modules for python, i was hopping those of you with experience could advice on the best approach if i wish to improve the performance of some segments of the code as much as possible.

As i understand, one can either write an extension using the python/numpy C-api, or wrap some already written pure C/C++/Fortran function to be called from the python code.

Naturally, tools like Cython are the easiest way to go, but i assume that writing the code by hand gives better control and provide better performance.

The question, and it may be to general, is which approach to use. Write a C or C++ extension? wrap external C/C++ functions or use callback to python functions?

I write this question after reading chapter 10 in Langtangen's "Python scripting for computational science" where there is a comparison of several methods to interface between python and C.

Upvotes: 0

Views: 1055

Answers (2)

DaveP
DaveP

Reputation: 7102

A good comparison of several different approaches can be found here. I have tried both cython, and wrapping my own fortran code using f2py. I found that f2py was the better way to go for my purposes. This was partly influenced by the fact that I understand fortran, but honestly modern dialects such as fortran 90 will look reasonably similar to python code using numpy and shouldn't be that hard to pick up.

With cython you start with the slow, pure python code, then you have to go through a tedious process of instrumenting your code, finding out where all the calls to the python API are, and entering the relevant cython keywords at the right places to turn it into faster C code. With fortran, you just write normal code and you are already getting full compiled speed without going a messy iterative process.

In addition, certain array operations in cython still result in slow calls to the Python API, particularly those involved in slice operations. In constrast, arrays in fortran are native types which the compiler understand and can optimise. Having said that, cython is advancing pretty rapidly, so this may change in the future.

The biggest downside I have found with f2py is that it doesn't support arrays of derived types (analogous to numpy's recarray). There was some hope that fwrap would be a replacement for f2py which would solve this issues, but it appears to be on the backburner at the moment. Incidentally it is based on cython.

Upvotes: 1

Félix Cantournet
Félix Cantournet

Reputation: 2001

I would say it depends on your skills/experience and your project. If this is very ponctual and you are profficient in C/C++ and you have already written python wrapper, then write your own extension and interface it.

If you are going to work with Numpy on other project, then go for the Numpy C-API, it's extensive and rather well documented but it is also quite a lot of documentation to process. At least I had a lot of difficulty processing it, but then again I suck at C.

If you're not really sure go Cython, far less time consuming and the performance are in most cases very good. (my choice) From my point of view you need to be a good C coder to do better than Cython with the 2 previous implementation, and it will be much more complexe and time consuming. So are you a great C coder ?

Also it might be worth your while to look into pycuda or some other GPGPU stuff if you're looking for performance, depending on your hardware of course.

Upvotes: 3

Related Questions