CODEWITHSUNDEEP
pythonrecursionmemorystackram
wa7d
wa7d

Reputation: 229

increasing recursion depth in python to 100000

python has a default maximum recursion depth which I am able to increase:

import sys

sys.setrecursionlimit(100000)

I'm using merge sort and when I try it on a list of 80000 elements, python "quits unexpectedly". This won't be an issue of I implemented merge sort iteratively, but I am interested in the recursive one.

I'm using a Mac OSX 8GB Memory. Is there a way to get this to work on my machine, or will it work on a better machine?

import sys

sys.setrecursionlimit(100000) # python has a recurison depth of < 1000~. so for the purpose of this assignment I'm increasing it

counter = 0


def merge_sort(lst):
    global counter
    if len(lst) <= 1:
        counter += 1   # increment counter when we divide array in two
        return lst
    mid = len(lst) // 2
    left = merge_sort(lst[:mid])
    right = merge_sort(lst[mid:])
    return merge(left, right)


def merge(left, right):
    global counter
    if not left:
        counter += 1   # increment counter when not left (not left - is also comparison)
        return right
    if not right:
        counter += 1   # the same as above for right
        return left
    if left[0] < right[0]:
        counter += 1   # and the final one increment
        return [left[0]] + merge(left[1:], right)
    return [right[0]] + merge(left, right[1:])


lines = [line.rstrip('\n') for line in open('words.txt')]

when I try the above on a 40000 it works and sorts the list:

print(merge_sort(lines[0:40000]))

but on 50000 or above it doesn't. The total number of words in the .txt file is around 80000

the message I get:

Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)

Upvotes: 1

Views: 5618

Answers (1)

UmNyobe
UmNyobe

Reputation: 22890

The problem comes from your merge(left, right) implementation which is recursive in O(n). You merge two sorted list by one element at each recursion step. The idea of merge being recursive may make sense in languages where tail-recursion is optimized, but it is not the case in python.

In general, merge is iterative as its complexity will always be at least the number of elements to merge. 

def merge(left, right):
    merged = []
    i = 0
    j = 0
    while i < len(left) and j < len(right) :
        if left[i] < right[j]:
            merged.append(left[i])
            i+=1
        else:
            merged.append(right[j])
            j+=1
    merged += left[i:]
    merged += right[j:]
    return merged

Upvotes: 4

Related Questions