CODEWITHSUNDEEP
python
Wang
Wang

Reputation: 8204

fast cumulative operation in python?

I have several operators f(a, b) like: "a*2+b*3", "a^b" ...

I would like to do something like:

a = a0
for _b in X:
    a = f(a, _b)

a0 will be N-byte value, X will be M*N byte buffer. In most case N <4, and a will not overflow. python reduce() will do the job, but for custom function it seems slow.

Can numexpr or other package speed up this?

Upvotes: 0

Views: 70

Answers (1)

u354356007
u354356007

Reputation: 3215

Here are some options:

  1. You may want to try functools.reduce solution on PyPy (just reduce if you're on Python 2). In addition you can use array instead of list.
  2. Use numpy.ufunc.reduce if your lists are big enough. It is written in C/Cython, but it takes longer to create small numpy array than a python stdlib's collection of the same size.

Update:

  1. Use Cython to implement f(a, b), reduce or both.

Pros: you can achieve almost C performance when it comes to dealing with numbers. You don't have to know Python C API.

Cons: almost C performance comes with writing almost C code. Use pointers, do some malloc, declare types explicitly, get a compatible C compiler. Before starting I would suggest to use cprofile in order to determine the critical code.

Upvotes: 1

Related Questions