CODEWITHSUNDEEP
pythonpandas
Jeremy
Jeremy

Reputation: 829

How can I normalize the data in a range of columns in my pandas dataframe

Suppose I have a pandas data frame surveyData:

I want to normalize the data in each column by performing:

surveyData_norm = (surveyData - surveyData.mean()) / (surveyData.max() - surveyData.min())

This would work fine if my data table only contained the columns I wanted to normalize. However, I have some columns containing string data preceding like:

Name  State  Gender  Age  Income  Height
Sam   CA     M        13   10000    70
Bob   AZ     M        21   25000    55
Tom   FL     M        30   100000   45

I only want to normalize the Age, Income, and Height columns but my above method does not work becuase of the string data in the name state and gender columns.

Upvotes: 32

Views: 63567

Answers (6)

Benjamin Ziepert
Benjamin Ziepert

Reputation: 1754

MinMax normalize all numeric columns with minmax_scale

import numpy as np
from sklearn.preprocessing import minmax_scale
# cols = ['Age', 'Height']
cols = df.select_dtypes(np.number).columns
df[cols] = minmax_scale(df[cols])

Note: Keeps index, column names or non-numerical variables unchanged.

Upvotes: 0

cwharland
cwharland

Reputation: 6703

You can perform operations on a sub set of rows or columns in pandas in a number of ways. One useful way is indexing:

# Assuming same lines from your example
cols_to_norm = ['Age','Height']
survey_data[cols_to_norm] = survey_data[cols_to_norm].apply(lambda x: (x - x.min()) / (x.max() - x.min()))

This will apply it to only the columns you desire and assign the result back to those columns. Alternatively you could set them to new, normalized columns and keep the originals if you want.

Upvotes: 42

seeker_after_truth
seeker_after_truth

Reputation: 497

I think it's really nice to use built-in functions

# Assuming same lines from your example
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
cols_to_norm = ['Age','Height']
survey_data[cols_to_norm] = scaler.fit_transform(survey_data[cols_to_norm])

Upvotes: 1

Gauravdeep
Gauravdeep

Reputation: 1

import pandas as pd
import numpy as np
# let Dataset here be your data#

from sklearn.preprocessing import MinMaxScaler
minmax = MinMaxScaler()

for x in dataset.columns[dataset.dtypes == 'int64']:
    Dataset[x] = minmax.fit_transform(np.array(Dataset[I]).reshape(-1,1))

Upvotes: -1

Yaron
Yaron

Reputation: 1852

I think it's better to use 'sklearn.preprocessing' in this case which can give us much more scaling options. The way of doing that in your case when using StandardScaler would be:

from sklearn.preprocessing import StandardScaler
cols_to_norm = ['Age','Height']
surveyData[cols_to_norm] = StandardScaler().fit_transform(surveyData[cols_to_norm])

Upvotes: 11

Alvaro Silvino
Alvaro Silvino

Reputation: 9733

Simple way and way more efficient:
Pre-calculate the mean:
dropna() avoid missing data.

mean_age = survey_data.Age.dropna().mean()
max_age = survey_data.Age.dropna().max()
min_age = survey_data.Age.dropna().min()

dataframe['Age'] = dataframe['Age'].apply(lambda x: (x - mean_age ) / (max_age -min_age ))

this way will work...

Upvotes: 4

Related Questions