CODEWITHSUNDEEP
python-3.xpandasdataframespecial-characters
question.it
question.it

Reputation: 2978

How to find special characters from Python Data frame

I need to find special characters from entire dataframe.

In below data frame some columns contains special characters, how to find the which columns contains special characters?

enter image description here

Want to display text for each columns if it contains special characters.

Upvotes: 4

Views: 28619

Answers (3)

Indrajit Bhatkar
Indrajit Bhatkar

Reputation: 11

# Whitespaces also could be considered in some cases.

import string

unwanted = string.ascii_letters + string.punctuation + string.whitespace
print(unwanted)

# This helped me extract '10' from '10+ years'.

df.col = df.col.str.strip(unwanted)

Upvotes: 1

Plinus
Plinus

Reputation: 318

To remove unwanted characters from dataframe columns, use regex:

def strip_character(dataCol):
    r = re.compile(r'[^a-zA-Z !@#$%&*_+-=|\:";<>,./()[\]{}\']')
    return r.sub('', dataCol)

df[resultCol] = df[dataCol].apply(strip_character)

Upvotes: 7

rafaelc
rafaelc

Reputation: 59274

You can setup an alphabet of valid characters, for example

import string
alphabet = string.ascii_letters+string.punctuation

Which is

'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

And just use 

df.col.str.strip(alphabet).astype(bool).any()

For example,

df = pd.DataFrame({'col1':['abc', 'hello?'], 'col2': ['ÃÉG', 'Ç']})


    col1    col2
0   abc     ÃÉG
1   hello?  Ç

Then, with the above alphabet,

df.col1.str.strip(alphabet).astype(bool).any()
False
df.col2.str.strip(alphabet).astype(bool).any()
True

The statement special characters can be very tricky, because it depends on your interpretation. For example, you might or might not consider # to be a special character. Also, some languages (such as Portuguese) may have chars like ã and é but others (such as English) will not.

Upvotes: 9

Related Questions