CODEWITHSUNDEEP
pythonscikit-learn
user139014
user139014

Reputation: 1495

parallelization with n_jobs in scikit-learn pairwise_distances

Thanks to Philip Cloud's great answer to a previous question, I went and dug into the source code for pairwise_distances in scikit.

The relevant part is:

def pairwise_distances(X, Y=None, metric="euclidean", n_jobs=1, **kwds):
    if metric == "precomputed":
        return X
    elif metric in PAIRWISE_DISTANCE_FUNCTIONS:
        func = PAIRWISE_DISTANCE_FUNCTIONS[metric]
        if n_jobs == 1:
            return func(X, Y, **kwds)
        else:
            return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
    elif callable(metric):
        # Check matrices first (this is usually done by the metric).
        X, Y = check_pairwise_arrays(X, Y)
        n_x, n_y = X.shape[0], Y.shape[0]
        # Calculate distance for each element in X and Y.
        # FIXME: can use n_jobs here too
        D = np.zeros((n_x, n_y), dtype='float')
        for i in range(n_x):
            start = 0
            if X is Y:
                start = i
            for j in range(start, n_y):
                # distance assumed to be symmetric.
                D[i][j] = metric(X[i], Y[j], **kwds)
                if X is Y:
                    D[j][i] = D[i][j]
        return D

Is it correct to understand from this that if I were to calculate a pairwise distance matrix like:

matrix = pairwise_distances(foo, metric=lambda u,v: haversine(u,v), n_jobs= -1)

where haversine(u,v) is a function that calculates the Haversine distance between two points and this function is not in PAIRWISE_DISTANCE_FUNCTIONS, that calculation would not be parallelized even though n_jobs= -1?

I realize that the #FIXME comment seems to imply this, but I want to make sure I'm not crazy, as it seems a little odd that there would be no informative alert thrown stating that the computation would not actually be parallelized when you pass n_jobs= -1 with a callable function that is not in PAIRWISE_DISTANCE_FUNCTIONS.

Upvotes: 2

Views: 1517

Answers (1)

user139014
user139014

Reputation: 1495

Confirmed. Passing a callable as the metric along with n_jobs= -1 will not result in parallelization if the callable is not in PAIRWISE_DISTANCE_FUNCTIONS.

Upvotes: 4

Related Questions