CODEWITHSUNDEEP
pythontensorflowmachine-learningkerasdeep-learning
J. Kim
J. Kim

Reputation: 105

What Activation Function is appropriate for input range (0,1) and output range (-∞,∞) for Regression Nerwork in Keras

input images are regularized to (0, 1) and output is float32 values having pseudo gaussian distribution (-∞,∞)

when fitted, both train and validation accuracy says over 0.999 but when predict using train and validation set, it does not reproduce itself.

predicted output shows only negative values( and few positive identical values )

is this problem caused by wrong selection of activation function?

i have tried, instead of 'relu', 'linear', 'sigmoid' too. the results was same.

model = Sequential()
model.add(Convolution1D(filters=64, kernel_size=2, input_shape=(img_width, img_height)))
model.add(Activation("relu"))
model.add(MaxPooling1D(pool_size=(2)))

model.add(Convolution1D(filters=32, kernel_size=2))
model.add(Activation("relu"))
model.add(MaxPooling1D(pool_size=(2)))

model.add(Flatten())
model.add(Dense(256))
model.add(Activation("relu"))
model.add(Dense(1, activation='linear'))

model.compile(loss='mse', optimizer=optimizers.RMSprop(lr=0.0001), metrics=['accuracy'])

Prediction done like this,

model.fit(x_train, y_train, epochs=2,
          validation_data=(x_valid, y_valid),
          batch_size=2048,
          shuffle='batch',
          use_multiprocessing=True)
# right after fitting 
result = model.predict(x_train, use_multiprocessing=True)

Upvotes: 1

Views: 1421

Answers (3)

Zabir Al Nazi Nabil
Zabir Al Nazi Nabil

Reputation: 11198

First of all, it's extremely hard to design a model to output in such a big range, the error rate of the model will be extremely high.

  1. I suggest you normalize your outputs in range (0., 1.) and use sigmoid in the last layer.

You can always use an inverse transform to reconstruct the original outputs.

mn = np.min(y_train)
mx = np.max(y_train)
y_train = (y_train - mn)/(mx - mn)

# ... train

# inverse transform
y_train_original = y_train*(mx-mn) + mn

when fitted, both train and validation accuracy says over 0.999 but when predict using train and validation set, it does not reproduce itself.

reason: overfitting. your data is impossible to learn with such complex output distribution, so the model just blindly memorizes the training data without learning any patterns.

to avoid :

  • use output normalizing.

  • model.add(Dense(256)) - reduce number of neurons here, try with 32->64->128

  • use dropout

Upvotes: 1

tensordude
tensordude

Reputation: 59

when fitted, both train and validation accuracy says over 0.999 but when predict using train and validation set, it does not reproduce itself.

This suggests something is going wrong with your prediction code, which you have not included. Either something wrong with your testing data or the way you are predicting (not loading weights?)

Upvotes: 1

Marco Cerliani
Marco Cerliani

Reputation: 22031

Convolution1D are not the standard choice to deal with images, I suggest you Convolution2D

Secondly, 'accuracy' is not the correct metric for regression task, good choice are mean squared error (mse), mean absolute error (mae), root mean squared error (rmse)

Upvotes: 1

Related Questions