CODEWITHSUNDEEP
pythonarraysnumpyoptimizationnumba
Laeneven
Laeneven

Reputation: 61

Numba - does nopython mode support list of tuples?

I would like to clarify that this is my first experience with Numba, so I am very far from being an expert. I am trying to manually implement an easy KNN, here's the code:

@jit(nopython=True)
def knn(training_set, test_set):
for q in range(len(test_set)):
    indexes = [-1]
    values = [np.inf]
    thres = values[-1]

    for u in range(len(training_set)):
        dist = 0
        flag = False
        dist = knn_dist(training_set[u], test_set[q], thres)
        if dist == 0:
            flag = True
        if not flag:

            '''
            Binary search to obtain the index
            '''    

            # Various code

return

Now, I would like to use the nopython mode of numba to optimize the code, here's part of the error:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
 in _call_incref_decref(self, builder, root_type, typ, value, funcname, getters)
    185             try:
--> 186                 meminfo = data_model.get_nrt_meminfo(builder, value)
    187             except NotImplementedError as e:

 in get_nrt_meminfo(self, builder, value)
    328                 raise NotImplementedError(
--> 329                     "unsupported nested memory-managed object")
    330         return value

NotImplementedError: unsupported nested memory-managed object

Both the training and the test set are lists of lists of tuples, I wonder if this data structure is supported by nopython, and if not (as it seems), which data structure could I use to make it possible? Am I forced to change numba mode?

P.s. to better clarify, an example of training/test is the following:

[[(0, 1), (1, 1), (2, 1), (3, 2), (4, 5)], [(0, 2), (1, 4), (2, 3), (3, 4), (4, 2)], [(0, 5), (1, 4), (2, 3), (3, 4), (4, 2)], [(0, 6), (1, 5), (2, 4), (3, 3), (4, 2)], [(0, 0), (1, 9), (2, 8), (3, 9), (4, 8)], [(0, 5), (1, 4), (2, 3), (3, 4), (4, 2)]]

Upvotes: 5

Views: 3956

Answers (1)

jpp
jpp

Reputation: 164753

does nopython mode support list of tuples?

Yes, it does. But, as your error message implies, not a nested list.

Am I forced to change numba mode?

No, you are not.

You can trivially convert your list of tuples L to a regular NumPy array:

L_arr = np.array(L)

Here's a demonstration and how you can test for yourself:

from numba import jit

L = [[(0, 1), (1, 1), (2, 1), (3, 2), (4, 5)], [(0, 2), (1, 4), (2, 3), (3, 4), (4, 2)],
     [(0, 5), (1, 4), (2, 3), (3, 4), (4, 2)], [(0, 6), (1, 5), (2, 4), (3, 3), (4, 2)],
     [(0, 0), (1, 9), (2, 8), (3, 9), (4, 8)], [(0, 5), (1, 4), (2, 3), (3, 4), (4, 2)]]

L_arr = np.array(L)

@jit(nopython=True)
def foo(x):
    return x

With L this gives an error:

print(foo(L))

LoweringError: Failed at nopython (nopython mode backend)
reflected list(reflected list((int64 x 2))): unsupported nested memory-managed object

With L_arr, you have a 3-dimensional NumPy array of shape (6, 5, 2):

print(foo(L_arr))

array([[[0, 1],
        [1, 1],
        [2, 1],
        [3, 2],
        [4, 5]],
        ...
       [[0, 5],
        [1, 4],
        [2, 3],
        [3, 4],
        [4, 2]]])

You may then wish to refactor your logic to work more efficiently with the NumPy array instead of a nested list of tuples.

Upvotes: 5

Related Questions