CODEWITHSUNDEEP
pythonnumpyscipylinear-algebrasparse-matrix
Tural Gurbanov
Tural Gurbanov

Reputation: 742

Sparse matrix dot product keeping only N-max values per result row

I've got a very huge csr sparse matrix M. I want to get dot product of this matrix to itself (M.dot(M.T)) and keep only N max values per each row in the result matrix R. The problem is that dot product M.dot(M.T) raises MemoryError. So I created modified implementation of dot function, that looks like:

def dot_with_top(m1, m2, top=None):
    if top is not None and top > 0:
        res_rows = []
        for row_id in xrange(m1.shape[0]):

            row = m1[row_id]
            if row.nnz > 0:
                res_row = m1[row_id].dot(m2)
                if res_row.nnz > top:
                    args_ids = np.argsort(res_row.data)[-top:]
                    data = res_row.data[args_ids]
                    cols = res_row.indices[args_ids]
                    res_rows.append(csr_matrix((data, (np.zeros(top), cols)), shape=res_row.shape))
                else:
                    res_rows.append(res_row)
            else:
                res_rows.append(csr_matrix((1, m1.shape[0])))
        return sparse.vstack(res_rows, 'csr')
    return m1.dot(m2)

It works fine but it's a bit slow. Is it possible to make this calculation faster or maybe you know some already existing method that do it faster?

Upvotes: 4

Views: 334

Answers (1)

CoMartel
CoMartel

Reputation: 3591

You can implement your loop over the number of row in a function, and call this function with the multiprocessing.Pool() object. This will parallelize the execution of your loop and should add a nice speedup.

Example :

from multiprocessing import Pool

def f(row_id): 
# define here your function inside the loop
    return vstack(res_rows, 'csr')

if __name__ == '__main__':
    p = Pool(4) # if you have 4 cores in your processor
    p.map(f, xrange(m1.shape[0]))

source : https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

Note that some python-implemented function already use multiprocessing (common in numpy), so you should check your processor activity when your script is running before implementing this solution.

Upvotes: 1

Related Questions