CODEWITHSUNDEEP
pythonpython-3.xreducebigdata
gaurav1207
gaurav1207

Reputation: 703

reduce large dataset in python using reduce function

I was trying a simple reduce function on a dataset of 1000000 integers, Here is what I'm trying to do,

from functools import reduce
a = (x for x in range(1,1000000))
reduce(lambda x,y: x*y,a)

Problem is, it is taking a lot of time! is there a faster way to implement this size of dataset?

Upvotes: 2

Views: 1922

Answers (1)

Kasravnd
Kasravnd

Reputation: 107287

The main problem here is the size of your result that makes the calculation takes so log. Basically, python can handle numbers as big as your memory will allow which means it's only limited by your memory. And note that as the numbers grow bigger, the cost of all operations on them increases. And as far as I know the reduce() function is the most python way for this task. But still since you only want the factorial of a particular number you can just use builtin math.factorial() function : which is by far faster than reduce(), and that's because python's math module function are implemented in C.

In [52]: %timeit math.factorial(10000)
100 loops, best of 3: 2.67 ms per loop

In [53]: %timeit reduce(mul, range(1, 10000))
10 loops, best of 3: 20.6 ms per loop

Upvotes: 2

Related Questions