CODEWITHSUNDEEP
machine-learningnormalizationscalingdata-science
Chandra Sutrisno
Chandra Sutrisno

Reputation: 3

Data Science Scaling/Normalization real case

When do data pre-processing, it is suggested to do either scaling or normalization. It is easy to do it when you have data on your hand. You have all the data and can do it right away. But after the model built and run, does the first data that comes in need to be scaled or normalized? If it needed, it only one single row how to scale or normalize it? How do we know what is the min/max/mean/stdev from each feature? And how is the incoming data is the min/max/mean each feature?

Please advise

Upvotes: 0

Views: 902

Answers (2)

pyAddict
pyAddict

Reputation: 1732

First of all you should know when to use scaling and normalization.

  1. Scaling - scaling is nothing but to transform your features to comparable magnitudes.Let say if you have features like person's income and you noticed that some have value of order 10^3 and some have 10^6.Now if you model your problem with this features then algorithms like KNN, Ridge Regression will give higher weight to higher magnitude of such attributes.To prevent this you need to first scale your features.Min-Max scaler is one of the most used scaling.

  2. Mean Normalisation - If after examining the distribution of the feature and you found that feature is not centered around zero then for the algorithm like svm where objective function already assumes zero mean and same order variance, we could have problem in modeling.So here you should do Mean Normalisation.

  3. Standardization - For the algorithm like svm, neural network, logistic regression it is necessary to have a variance of the feature in the same order.So why don't we make it to one.So in standardization, we make the distribution of features to zero mean and unit variance.

Now let's try to answer your question in terms of training and testing set. So let's say you are training your model on 50k dataset and testing on 10k dataset. For the above three transformations, the standard approach says that you should fit any normalizer or scaler to only training dataset and use only transform for the testing dataset. In our case, if we want to use standardization then we will first fit our standardizer on 50k training dataset and then used to transform it 50k training dataset and also testing dataset.

Note - We shouldn't fit our standardizer to test dataset, in place of we will use already fitted standardizer to transform testing dataset.

Upvotes: 3

Dr. Snoopy
Dr. Snoopy

Reputation: 56357

Yes, you need to apply normalization to the input data, else the model will predict nonsense.

You also have to save the normalization coefficients that were used during training, or from training data. Then you have to apply the same coefficients to incoming data.

For example if you use min-max normalization:

f_n = (f - min(f)) / (max(f) - min_(f))

Then you need to save the min(f) and max(f) in order to perform normalization for new data.

Upvotes: 2

Related Questions