CODEWITHSUNDEEP
pythonscikit-learnnormalizationscaling
De Gninou
De Gninou

Reputation: 357

Standardize dataset containing too large values

I standardize features to mean=0 and sd=1 using preprocessing.scale astype('float64'). I got the following warning:

UserWarning: Numerical issues were encountered when centering the data and might not be solved. Dataset may contain too large values. You may need to prescale your features. warnings.warn("Numerical issues were encountered "

Here is a sample of the dataset: 

    col1    col2    col3    col4    col5    col6    col7    col8    col9    col10   col11   col12   col13
0   327 143.04  123.66  101.71  89.36575914 0.668110013 84.13713837 588.103818  633.6584113 525.5505746 132.966095  13.05099964 131.7220566
1   1010    188.98  176.78  137.33  89.36575914 0.620949984 40.52060699 1413.802012 3705.255352 1641.459378 106.3353716 7.69299984  472.4249759
2   1485    166.67  141.72  111.07  98.91169739 0.979290009 100 3580.441388 4327.644518 3242.16829  111.2140427 13.05300045 1164.119187
3   78  54.27   83.01   161.74  95.0061264  0.968744297 100 35644.07894 37765.71684 15667.95157 106.3043671 7.448999882 850.651571
4   591 132.86  121.22  108.13  103.231369  1.039739966 100 9348.743837 10699.19772 7144.242782 101.7313309 8.788999557 1382.113557
5   562 134.98  141.72  141.15  89.36575914 0.968744297 100 3046.147835 3710.575743 2716.801411 106.3353716 18.26099968 1076.131188
6   1030    110.83  79.08   50.87   89.36575914 0.952409983 97.35466766 11348.70932 11928.21847 7637.253514 102.3456802 9.793620323 1164.119187
7   534 109.06  109.14  106.12  89.36575914 0.968744297 100 43007.67453 54008.70819 29971.03064 106.3353716 5.602000237 1164.119187

What is prescaling ? An what are my options to do so ?

Upvotes: 6

Views: 9491

Answers (3)

whats up
whats up

Reputation: 1

If you want this for PCA, try this:

 df=your data frame
 dataForPCA=preprocessing.StandardScaler().fit_transform(df.T)

it will bring your data into the format needed for PCA by formatting the data to 0-1 and also transposing your df into an np array ready to use

if you don't want this for PCA :

 df=your data frame
 dataForPCA=preprocessing.StandardScaler().fit_transform(df)

Upvotes: -1

lqi
lqi

Reputation: 137

Converting the data frame to float type works for me:

df = df.astype(float)
scaled_df = preprocessing.scale(df)

Upvotes: 1

De Gninou
De Gninou

Reputation: 357

I solve the problem using StandardScaler and taking example on the following code as suggested here: 

from sklearn import preprocessing
# Get column names first
names = df.columns
# Create the Scaler object
scaler = preprocessing.StandardScaler()
# Fit your data on the scaler object
scaled_df = scaler.fit_transform(df)
scaled_df = pd.DataFrame(scaled_df, columns=names)

Upvotes: 9

Related Questions