CODEWITHSUNDEEP
pythonpandas
Christopher Wallace
Christopher Wallace

Reputation: 1

Pandas not replacing character in large dataframe

When I create a small test dataframe this code works fine, but when attempting to use it after importing a large excel file it is not substituting the character.

import pandas as pd
df = pd.DataFrame({'A':[1,2,3],
                    'B':[4,5,6],
                    'C':['`f;','d:','sda`sd'],
                    'D':['s','d;','d;p`'],
                    'E':[5,3,6],
                    'F':[7,4,3]})
df.replace({'`':''}, regex=True)

The results are as expected:

    A   B   C   D   E   F
0   1   4   f;  s   5   7
1   2   5   d:  d;  3   4
2   3   6   sdasd   d;p 6   3

However when I load up a large excel file:

import pandas as pd
excel_file = f'C:\testfile.xlsx'
df = pd.read_excel(excel_file,sheet_name='Details', dtype=str)
df.iloc[20831].loc['Group Number']

Result:

'008513L-0005 `'

Then run the replace:

df.replace({'`':''}, regex=True)
df.iloc[20831].loc['Group Number']

Result:

'008513L-0005 `'

Upvotes: 0

Views: 29

Answers (1)

Fatih Afşin
Fatih Afşin

Reputation: 11

We can solve the problem you encounter in Natural Language Processing Methods with the "string punctuation" function.

import string #We have defined the string library.
def remove_punctuation (text): #We form our function.
    no_punc = "".join([i for i in text if i not in string.punctuation])
    return no_punc
#We apply our function to the corresponding column of our data set.
df['C'] = df['C'].apply(lambda x: remove_punctuation (x))
df['D'] = df['D'].apply(lambda x: remove_punctuation (x))

enter image description here

Upvotes: 1

Related Questions