CODEWITHSUNDEEP
pythonarraysperformancenumpy
Varlor
Varlor

Reputation: 1471

Python: Compare Elements of two arrays

I want to compare elements of two numpy arrays and delete the elements of one of those arrays if the eucledean distance between the coordinates is smaller than 1 and the time is the same. data_CD4 and data_CD8 are the arrays. The elements of the arrays are lists with 3D Coordinates and the time as 4th element (numpy.array([[x,y,z,time],[x,y,z,time].....]). Co is the Cutoff, here 1.

for i in data_CD8:
        for m in data_CD4:
            if distance.euclidean(tuple(i[:3]),tuple(m[:3])) < co and i[3]==m[3] :
                data_CD8=np.delete(data_CD8, i, 0)

Is there a faster approach to do that? The first array has 5000 elements, the second 2000, so it tooks too much time.

Upvotes: 1

Views: 3136

Answers (3)

Daniel F
Daniel F

Reputation: 14399

This should be a vectorized method. 

mask1 = np.sum((data_CD4[:, None, :3] - data_CD8[None, :, :3])**2, axis = -1) < co**2
mask2 = data_CD4[:, None, 3] == data_CD8[None, :, 3]
mask3 = np.any(np.logical_and(mask1, mask2), axis = 0)
data_CD8 = data_CD8[~mask3]

mask1 should speed the distance calculations up as it doesn't require a square root call. mask1 and mask2 are 2-D arrays which we squeeze to 1d by np.any Doing all the deleting at the end prevents the pile of read/writes.

Speed testing:

a = np.random.randint(0, 10, (100, 3))

b = np.random.randint(0, 10, (100, 3))

%timeit cdist(a,b) < 5  #Divakar's answer
10000 loops, best of 3: 133 µs per loop

%timeit np.sum((a[None, :, :] - b[:, None, :]) ** 2, axis = -1) < 25  # My answer
1000 loops, best of 3: 418 µs per loop

And the C-compiled code wins, even when adding an unnecessary square root.

Upvotes: 2

Divakar
Divakar

Reputation: 221574

Here's a vectorized approach using Scipy's cdist -

from scipy.spatial import distance

# Get eucliden distances between first three cols off data_CD8 and data_CD4
dists = distance.cdist(data_CD8[:,:3], data_CD4[:,:3])

# Get mask of those distances that are within co distance. This sets up the 
# first condition requirement as posted in the loopy version of original code.
mask1 = dists < co

# Take the third column off the two input arrays that represent the time values.
# Get the equality between all time values off data_CD8 against all time values
# off data_CD4. This sets up the second conditional requirement.
# We are adding a new axis with None, so that NumPY broadcasting
# would let us do these comparisons in a vectorized manner.
mask2 = data_CD8[:,3,None] == data_CD4[:,3]

# Combine those two masks and look for any match correponding to any 
# element off data_CD4. Since the masks are setup such that second axis
# represents data_CD4, we need numpy.any along axis=1 on the combined mask.
# A final inversion of mask is needed as we are deleting the ones that 
# satisfy these requirements.
mask3 = ~((mask1 & mask2).any(1))

# Finally, using boolean indexing to select the valid rows off data_CD8
out = data_CD8[mask3]

Upvotes: 2

widerin
widerin

Reputation: 86

if you have to compare all items in data_CD4 to the items in data_CD8 while removing data from data_CD8 it might be better to make the second iterable smaller on each iteration, which of course depends on your most common case. 

for m in data_CD4:
    for i in data_CD8:
        if distance.euclidean(tuple(i[3:]),tuple(m[3:])) < co and i[3]==m[3] :
            data_CD8 = np.delete(data_CD8, i, 0)

Based on big O notation - and since this is O(n^2) - i don't see a faster solution.

Upvotes: 0

Related Questions