CODEWITHSUNDEEP
tensorflowneural-networkdeep-learningregression
python_tensorflow
python_tensorflow

Reputation: 349

Tensorflow regression data normalization before inputting into neural network

I have to perform a regression which maps 100 dimensional vector inputs to 100 dimensional vector outputs. Both, the inputs and outputs lie in the range of 0 to 1200. I am using 4 layer MLP (multi layer perceptron) built using tensorflow fully connected layers. As is commonly suggested, I normalized the inputs, so that they lie in the range 0-1. Thus, I divided the input data by the maximum value it attains. However, such an approach failed to produce any useful result (I did not normalize the outputs though). Thereafter, I did not normalize the inputs, and repeated the MLP experiment. This time, the neural network performed quite well. I think that due to normalization, all the activations were close to 0, and failed to "catch-up to the outputs". What am I missing here? Should we always normalize the inputs/ or not?

Upvotes: 1

Views: 1217

Answers (1)

amirbar
amirbar

Reputation: 839

In your specific application, the normalization you used made you lose valuable information. consider the following two inputs before normalization:

x1 = [2, ..., 10, ...] # 2 is min, 10 is max
x2 = 5*x1

you can see that max(x2) = 5*max(x1), thus, after normalization both samples are equal.

So, in your case, it seems like the scale information is important. losing it resulted in poor results. In different applications (image processing for example) this may not be the case and such normalization will help to make the learning process more smooth.

Why is the scale important in your application? I'm not sure. But I can give you an example from the medical imaging domain. In CT scans, every pixel value corresponds to a HU value, which is essentially a metric of density in that specific location. Assuming we have two images with a similar pattern like the samples on top, x1 can correspond to a an image mainly containing soft tissue and x2 bone. Do we want normalize both to the same sample? probably not.

I think the link below is a good resource to further read about normalization/standardization:

http://sebastianraschka.com/Articles/2014_about_feature_scaling.html#about-standardization

Upvotes: 1

Related Questions