CODEWITHSUNDEEP
machine-learningkerasneural-networkconv-neural-networkautoencoder
Randy Vogel
Randy Vogel

Reputation: 425

Keras autoencoder negative loss and val_loss with data in range [-1 1]

I am trying to adapt keras autoencoder example to a my data. I have the following network:

Xtrain = np.reshape(Xtrain, (len(Xtrain), 28, 28, 2))
Xtest = np.reshape(Xtest, (len(Xtest), 28, 28, 2))

input_signal = Input(shape=(28, 28, 2))
x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_signal)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same', name='encoder')(x)

# added Dense layers, is that correct?
encoded2 = Flatten()(encoded)
encoded2 = Dense(128, activation='sigmoid')(encoded2)
encoded2 = Dense(128, activation='softmax')(encoded2)
encoded3 = Reshape((4, 4, 8))(encoded2)

x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded3)
x = UpSampling2D((2, 2))(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(16, (3, 3), activation='relu')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(2, (3, 3), activation='sigmoid', padding='same')(x)

autoencoder = Model(inputs=input_signal, outputs=decoded)
encoder = Model(input_signal, encoded2)
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
autoencoder.fit(Xtrain, Xtrain, epochs=100, batch_size=128, shuffle=True, validation_data=(Xtest, Xtest))

And, when I'm running on MNIST data, which are normalized to [0,1] everything works fine, but with my data that are in range [-1,1] I only see negative losses and 0.0000 accuracy while training. If I do data = np.abs(data), training starts and looks that goes well, but doing abs() on data makes no reasons to train data fakes.

The data I'm trying to feed to network are IQ channels of signal, 1st channel for real part, and 2nd channel for imag part, so both are normalized to a [-1 1], and both often contains very low values, e.g. 5e-12. I have shaped them to a (28,28,2) input.

I have also added Dense layers in the middle of autoencoder, as I wish to make predictions about classes (that are fitted automatically) when autoencoder completes training. Am I did this correctly, does this breaks the network?

Upvotes: 2

Views: 6304

Answers (2)

desertnaut
desertnaut

Reputation: 60370

There are several issues with your question, including your understanding of autoencoders and their usage. I strongly suggest at least going through the Keras blog post Building Autoencoders in Keras (if you do have gone through it, arguably you have to do it again, this time more thoroughly).

A few general points, most of which are included in the above linked post:

  1. Autoencoders are not used for classification, hence it makes no sense to ask for a metric such as accuracy. Similarly, since the fitting objective is the reconstruction of their input, categorical cross entropy is not the correct loss function to use (try binary cross entropy instead).
  2. The very existence of the intermediate dense layers you use is puzzling, and even more puzzling is the choice of a sigmoid layer followed by a softmax one; the same holds for the sigmoid choice in your final, decoded layer. Both these activation functions are normally used for classification purposes at final layers, so again refer to point (1) above.
  3. I strongly suggest you start with a model demonstrated in the blog post linked above, and, if necessary, incrementally modify it to fit your purpose, as I am not sure what you have built here can even qualify as an autoencoder in the first place.

Upvotes: 1

sladomic
sladomic

Reputation: 876

You are mixing between binary ('sigmoid') and categorical ('softmax' and 'categorical_crossentropy'). Change the following:

  1. Remove the dense layers in between and feed 'encoded' instead of 'encoded3' to the decoder
  2. Change the autoencoder loss to 'binary_crossentropy'

Alternatively if you really want to try the dense layers in between, just use them without an activation function (None)

Upvotes: 0

Related Questions