CODEWITHSUNDEEP
pythontensorflowkeras
Tabin
Tabin

Reputation: 149

Machine Learning with Keras: Different Validation Loss for the Same Model

I am trying to use keras to train a simple feedforward network. I tried two different methods of what I think is the same network, but one is performing significantly better. The first one and the better performing one is the following:

inputs = keras.Input(shape=(384,))
dense = layers.Dense(64, activation="relu")
x = dense(inputs)
x = layers.Dense(64, activation="relu")(x)
outputs = layers.Dense(384)(x)

model = keras.Model(inputs=inputs, outputs=outputs, name="simple_model")

model.compile(loss='mse',optimizer='Adam')

history = model.fit(X_train, 
                    y_train_tf, 
                    epochs=20, 
                    validation_data=(X_test, y_test), 
                    steps_per_epoch=100, 
                    validation_steps=50)

and it settles on a validation loss of about 0.2. The second model performs much worse:

model = keras.models.Sequential()

model.add(Dense(64, input_shape=(384,), activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(384, activation='relu'))

optimizer = tf.keras.optimizers.Adam()

model.compile(loss='mse', optimizer=optimizer)

history = model.fit(X_train, 
                    y_train_tf, 
                    epochs=20, 
                    validation_data=(X_test, y_test), 
                    steps_per_epoch=100, 
                    validation_steps=50)

and this has validation loss of around 5. But when I do model.summary, they look virtually the same. Is there something wrong with the second model?

Upvotes: 0

Views: 47

Answers (1)

Dmitry Petrov
Dmitry Petrov

Reputation: 120

I am not sure that they are the same since second model has relu activation after last layer (384 units) and first doesn't. This might be the issue since default activation of the Keras dense layer is None.

Upvotes: 1

Related Questions