CODEWITHSUNDEEP
pythonpandas
SantoshGupta7
SantoshGupta7

Reputation: 6197

How to perform a conditional on a column of lists (considering each item in the list) in Pandas

Say that I have a column of lists. If the list has at least one item in a set, I want to keep the row, otherwise I want to drop the row.

Here is a minimal example

#create the df 
d={'range':list(range(0,3))}
df=pd.DataFrame(d)
l=[1, 2, 3]
m =[4, 5, 6]
n =[1, 7, 8]
df['var_list']=''
df['var_list'][0]=l
df['var_list'][1]=m
df['var_list'][2]=n
df.head(3)

result

range   var_list
0   0   [1, 2, 3]
1   1   [4, 5, 6]
2   2   [1, 7, 8]

And this is the set I want to use

setS = {1, 2}

What I'm trying to do is, if any row's list has an item that's in the set, keep that row, otherwise drop that row.

So this is the desired result:

range   var_list
0   0   [1, 2, 3]
2   2   [1, 7, 8]

What I tried

df2 = df[df['var_list'].isin(setS)]

This is the error I got 

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: unhashable type: 'list'

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
<ipython-input-56-90ea3b42ebf3> in <module>()
----> 1 df2 = df[df['var_list'].isin(setS)]

2 frames
/usr/local/lib/python3.6/dist-packages/pandas/core/series.py in isin(self, values)
   4512         Name: animal, dtype: bool
   4513         """
-> 4514         result = algorithms.isin(self, values)
   4515         return self._constructor(result, index=self.index).__finalize__(self)
   4516 

/usr/local/lib/python3.6/dist-packages/pandas/core/algorithms.py in isin(comps, values)
    478             comps = comps.astype(object)
    479 
--> 480     return f(comps, values)
    481 
    482 

/usr/local/lib/python3.6/dist-packages/pandas/core/algorithms.py in <lambda>(x, y)
    454 
    455     # faster for larger cases to use np.in1d
--> 456     f = lambda x, y: htable.ismember_object(x, values)
    457 
    458     # GH16012

pandas/_libs/hashtable_func_helper.pxi in pandas._libs.hashtable.ismember_object()

SystemError: <built-in method view of numpy.ndarray object at 0x7fcc893844e0> returned a result with an error set

Upvotes: 4

Views: 580

Answers (3)

Andy L.
Andy L.

Reputation: 25259

List comprehension with python set intersection to create the mask and slice

m = [len(setS & x) > 0 for x in df.var_list.map(set)]
df[m]

Out[21]:
   range   var_list
0      0  [1, 2, 3]
2      2  [1, 7, 8]

Upvotes: 1

oppressionslayer
oppressionslayer

Reputation: 7224

You can accomplish this with an apply map and or operator, by converting the list to a set and comparing:

[df.var_list.apply(lambda x: False if len(setS | set(x)) > 4 else True)]                                                                                                          
Out[3343]: 
   range   var_list
0      0  [1, 2, 3]
2      2  [1, 7, 8]

Upvotes: 0

Henry Yik
Henry Yik

Reputation: 22503

A column of lists is not how pandas normally works. You have to check for the items in the list explicitly:

print (df[df["var_list"].transform(lambda x: bool(set(x)&sets))])

#
   range   var_list
0      0  [1, 2, 3]
2      2  [1, 7, 8]

Upvotes: 1

Related Questions