CODEWITHSUNDEEP
pythonpandasdataframe
Selah
Selah

Reputation: 8054

Pandas dataframe - identify rows with value over threshold in any column

I have a word similarity matrix stored as a pandas data-frame where the columns are a "seed set" of ~400 words and the row indexes are a large dictionary of ~50,000 words. The value at any row/column is the similarity from 0 to 1 between the two words.

>>> df_sim_mf.info()
<class 'pandas.core.frame.DataFrame'>
Index: 46265 entries, #angry to wonga
Columns: 451 entries, abandon to wrongs
dtypes: float64(451)
memory usage: 159.5+ MB
>>> df_sim_mf.sample(10).sample(5, axis = 1)
              nationality    purest     unite   lawless      riot
assaulted        0.114270 -0.140504  0.182024  0.434651  0.510618
peekaboo        -0.008734 -0.027742  0.051084  0.260245  0.201117
antibiotic       0.145310  0.270748 -0.126459 -0.083965  0.043086
killin          -0.102474  0.123550  0.055935 -0.115381  0.285997
warrior          0.005229  0.281967  0.261230  0.344130  0.359228
actionscript    -0.029405  0.077793  0.114047 -0.052599 -0.123401
controversy      0.336688  0.271007  0.373474  0.362565  0.305548
nic              0.164550 -0.159097  0.080056  0.271184  0.231357
healy            0.072831  0.102996  0.286538  0.335697  0.183730
uncovered        0.061310  0.274003  0.328383  0.300315  0.277491

I'm trying to find all words from my large dictionary that are within a certain similarity range from ANY of my "seed set". That is, I'd like to select every row that contains at least one value over 0.75.

Can I do this with a few simple pandas commands?

Upvotes: 4

Views: 2304

Answers (1)

ldirer
ldirer

Reputation: 6756

You could do:

df.loc[(df > 0.75).sum(axis=1) > 0, :]

and get the index attribute if you just want the words.

Upvotes: 3

Related Questions