CODEWITHSUNDEEP
pythontensorflowmachine-learningkerasscikit-learn
Sasha1296
Sasha1296

Reputation: 179

Keras model gives same output for all inputs

I am making a neural network that is supposed to take 480 data points and output 18 data points. The inputs are magnetic field strengths and the outputs are coordinates of objects detected(will be zero if no object is detected) so no data point is really categorical. For some reason, when I train the model I get the same output for every input I try, for example:

>>> output2 = loaded_model.predict(X_)
>>> output2[0]
array([0.32035217, 0.3027814 , 0.2977892 , 0.30922157, 0.3294088 ,
       0.40853357, 0.09848618, 0.15266985, 0.29188123, 0.31177315,
       0.4652696 , 0.6406114 , 0.204305  , 0.23156416, 0.19870688,
       0.21269864, 0.28510743, 0.29115945], dtype=float32)
>>> output2[100]
array([0.32035217, 0.3027814 , 0.2977892 , 0.30922157, 0.3294088 ,
       0.40853357, 0.09848618, 0.15266985, 0.29188123, 0.31177315,
       0.4652696 , 0.6406114 , 0.204305  , 0.23156416, 0.19870688,
       0.21269864, 0.28510743, 0.29115945], dtype=float32)

The code I used to generate this model is:

import pandas as pd
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from numpy import savetxt
from keras.optimizers import Adam



from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras import regularizers
from keras.layers.advanced_activations import LeakyReLU



from keras.models import model_from_json
#from keras.layers import LeakyReLU
import matplotlib.pyplot as plt



df = pd.read_csv("C:/Users/an/Desktop/python processing/try_2/Hx_output.csv")
dataset = df.values
X = dataset[:,0:480]
Y = dataset[:,480:499]


min_max_scaler = preprocessing.MinMaxScaler()
X_scale = min_max_scaler.fit_transform(X)
Y_scale = min_max_scaler.fit_transform(Y)


X_train, X_val_and_test, Y_train, Y_val_and_test = train_test_split(X_scale, Y_scale, test_size=0.3)
X_val, X_test, Y_val, Y_test = train_test_split(X_val_and_test, Y_val_and_test, test_size=0.5)



pd.DataFrame(X_val).to_csv("C:/Users/an/Desktop/python processing/try_2/X_val.csv", header=None, index=None)
pd.DataFrame(Y_val).to_csv("C:/Users/an/Desktop/python processing/try_2/Y_val.csv", header=None, index=None)




# activation = LeakyReLU(alpha=0.05)
model = Sequential([    Dense(480, activation= 'sigmoid', kernel_regularizer=regularizers.l2(0.01), input_shape=(480,)),
                        Dropout(0.3),
                        Dense(5000, activation= 'softplus', kernel_regularizer=regularizers.l2(0.01)),
                        Dropout(0.3),
                        Dense(5000, activation= 'softplus', kernel_regularizer=regularizers.l2(0.01)),
                        Dropout(0.3),
                        Dense(5000, activation= 'softplus', kernel_regularizer=regularizers.l2(0.01)),
                        Dropout(0.3),
                        Dense(5000, activation= 'softplus', kernel_regularizer=regularizers.l2(0.01)),
                        Dropout(0.3),
                        Dense(5000, activation= 'softplus', kernel_regularizer=regularizers.l2(0.01)),
                        Dropout(0.3),
                        Dense(18, activation= 'sigmoid', kernel_regularizer=regularizers.l2(0.01))])





#opt = keras.optimizers.Adam(lr=0.001)
model.compile(optimizer= Adam(lr=0.0001),              loss='mean_squared_error',              metrics=['mean_squared_error'])
##callbacks=[early_stopping_monitor]
hist = model.fit(X_train, Y_train,          batch_size=32, epochs=150,          validation_data=(X_val, Y_val))

print("Done training !!!")



# serialize model to JSON
model_json = model.to_json()
with open("C:/Users/an/Desktop/python processing/try_2/model.json", "w") as json_file:
    json_file.write(model_json)
# serialize weights to HDF5
model.save_weights("C:/Users/an/Desktop/python processing/try_2/model.h5")
print("Saved model to disk")

plt.plot(hist.history['mean_squared_error'])
#plt.plot(hist.history['val_acc'])
plt.title('Model Mean Squared Error')
plt.ylabel('MeanSquaredError')
plt.xlabel('Epoch')
#plt.legend(['Train', 'Val'], loc='lower right')
plt.show()

I read that some of the causes of this are having a learning rate that is too high, disabling the "trainable" feature of layers, having a small batch size. I tried to decrease my learning rate to 0.0001 and I still got the same result, as far as I can tell all of my layers are trainable and the last thing that could be the problem which is batch size I have not tried yet. I have a couple thousand training samples so maybe this is the problem and am increasing it from 32 to 400 for the new round of training I will do soon, but maybe the issue is somewhere else that I am not seeing?

Also I read it's a good idea to use callbacks=['early_stopping_monitor'] is it appropriate in this case?

Edit: Also could the kernel_regularizer=regularizers.l2(0.01) term have an effect on this?

Upvotes: 4

Views: 4814

Answers (2)

Abhishek Verma
Abhishek Verma

Reputation: 1729

You are overfitting because of the huge amount of neurons you are using in your Dense layers. Please reduce them to a reasonable range [512, 1024] sticking to the lower side and then, moving up if needed. Reduce the number of layers also, increase only if necessary.

Upvotes: 4

Pranjal dubey
Pranjal dubey

Reputation: 93

Are you sure that your loading you model's weights too. Try

model.load_weights(path)

If you don't load your weights you obtained after training then the model will use random weights and give same prediction every time since the model hasn't learnt anything.

Upvotes: 0

Related Questions