CODEWITHSUNDEEP
pythoncnumpyopenmpcython
ffi23
ffi23

Reputation: 121

Multithread prange loop throws "double free or corruption (fasttop)" error

I made a few changes to the original question. Turns out the malloc part is actually probably the issue, as suggested in the comments.

I want to run a function in a Cython prange loop as in the code below. This code throws a "double free or corruption (fasttop)" error.

When I run the code with the prange flag "num_threads=1" all is fine. I understand my code is not thread safe probably, but I don't understand why.

import numpy as np
cimport numpy as np
cimport cython
from cython.parallel import prange
from libc.stdlib cimport malloc, free

cdef int my_func(int[:] arr_cy, int c) nogil except -1:

    cdef int i
    cdef int *arr_to_process = <int *>malloc(c * sizeof(int))
    if not arr_to_process:
        with gil:
            raise MemoryError()
    try:
        for i in range(c):
            arr_to_process[i] = 1
    finally:
        free(arr_to_process)
    return 0

def going(a):
    cdef int c 
    for c in prange(100000, nogil=True, num_threads=2):
        my_func(a, c)

def get_going(iterations):
    arr = np.arange(1000000, dtype=np.intc)
    cdef int [:] arr_v = arr

    for a in range(iterations):
        print('iter %i' %a)
        going(arr_v)

if I run get_going(iterations) with enough iterations, like 30, this always throws an error. I feel I am being very dumb but I don't get it. Thank you for your help.

Upvotes: 0

Views: 872

Answers (2)

ffi23
ffi23

Reputation: 121

The answer by @DavidW is ok, but it is not a full answer to the problem. After a bit of fiddling I found what I was looking for: I needed to use pointers for the memoryviews, as explained in the Pass data from a C function via pointer section on the cython docs. Here is the full working code.

cdef int my_func(int arr_cy[], int c) nogil except -1:

    cdef int i
    cdef int *arr_to_process = <int *>malloc(c * sizeof(int))
    if not arr_to_process:
        with gil:
            raise MemoryError()
    try:
        for i in range(c):
            arr_to_process[i] = 1
    finally:
        free(arr_to_process)
    return 0

def going(a):
    cdef int c
    cdef int [:1] arr_v = a
    for c in prange(100000, nogil=True, num_threads=2):
        my_func(&arr_v[0], c)

def get_going(it):
    arr = np.arange(1000000, dtype=np.intc)

    for ii in range(it):
        print('iter %i' %ii)
        going(arr)

Upvotes: 0

DavidW
DavidW

Reputation: 30915

I originally identified one issue that wasn't causing your problem but did need fixing (this is now fixed in the edited code): Cython has no way of knowing that the exception has been raised - in the C API an exception is indicated by returning NULL, but your function is void. See the relevant bit of the documentation. You have two options: define the function with except * to always check for an exception, or define it with an error code:

    cdef int my_func(int[:] arr_cy, int c) nogil except 1:
        # ... code goes here
        return 0 # indicate no error

Cython will automatically use this when you raise an exception.

The actual issue is the line my_func(a, c). The conversion from a Numpy array to a memoryview does need some kind of locking (i.e. the GIL) or there's a race condition with reference counting. This race condition causes it to be freed when it shouldn't, hence the error

The solution is to generate a memoryview of a outside the loop:

cdef int[:] a_mview = a
# then inside the prange loop
     my_func(a_mview, c).

Use of the memoryview is fine within the parallel section but it's just the initial creation that's a problem. I think Cython not flagging this as an error when you compile is a bug, and might be worth reporting.

Upvotes: 1

Related Questions