CODEWITHSUNDEEP
pythonpandas
user10671234
user10671234

Reputation: 67

Remove any empty fields in a loop?

A list has many paths of certain csv's. How to check if each csv in every loop has any empty columns and delete them if they are.

Code:

for i in list1:
    if (list1.columns = '').any():
        i.remove that column

Hope this explains what I am talking about.

Upvotes: 3

Views: 276

Answers (1)

jezrael
jezrael

Reputation: 862761

Sample:

df = pd.DataFrame({
        '':list('abcdef'),
         'B':[4,5,4,5,5,np.nan],
         'C':[''] * 6,
         'D':[np.nan] * 6,
         'E':[5,3,6,9,2,4],
         'F':list('aaabb') + ['']
})

print (df)
        B C   D  E  F
0  a  4.0   NaN  5  a
1  b  5.0   NaN  3  a
2  c  4.0   NaN  6  a
3  d  5.0   NaN  9  b
4  e  5.0   NaN  2  b
5  f  NaN   NaN  4

Removed first column, because empty column name - it means filtering only columns with no empty values with loc and boolean indexing:

df1 = df.loc[:, df.columns != '']
print (df1)
     B C   D  E  F
0  4.0   NaN  5  a
1  5.0   NaN  3  a
2  4.0   NaN  6  a
3  5.0   NaN  9  b
4  5.0   NaN  2  b
5  NaN   NaN  4

Reoved column C, because filled only empty values - compare all values if not empty values and get at least one True per column by DataFrame.any, also filter by boolean indexing with loc:

df2 = df.loc[:, (df != '').any()]
print (df2)
        B   D  E
0  a  4.0 NaN  5
1  b  5.0 NaN  3
2  c  4.0 NaN  6
3  d  5.0 NaN  9
4  e  5.0 NaN  2
5  f  NaN NaN  4

print ((df != ''))
            B      C     D     E      F
0  True  True  False  True  True   True
1  True  True  False  True  True   True
2  True  True  False  True  True   True
3  True  True  False  True  True   True
4  True  True  False  True  True   True
5  True  True  False  True  True  False

print ((df != '').any())
      True
B     True
C    False
D     True
E     True
F     True
dtype: bool

Removed column D because filled only missing values with function dropna:

df3 = df.dropna(axis=1, how='all')
print (df3)
        B C  E  F
0  a  4.0    5  a
1  b  5.0    3  a
2  c  4.0    6  a
3  d  5.0    9  b
4  e  5.0    2  b
5  f  NaN    4

Upvotes: 3

Related Questions