CODEWITHSUNDEEP
pythonkerasneural-networkautoencoder
Magellan88
Magellan88

Reputation: 2573

Which parameters prohibit the autoencoder from learning the right distribution?

I have a very minimal example of an autoencoder:

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

First I create a data set with the highly correlated variables A and B (that are already normalized)

X = pd.DataFrame( (np.random.randn(1000,2)), columns=["A", "B"] )
X["B"] = X["A"] + X["B"]/4

Then I setup the autoencoder and train it

aeInput = Input(2)
encode = Dense(2, activation='relu')(aeInput)
aeOutput = Dense(2, activation='relu')(encode)

AE = Model(aeInput, aeOutput, name="autoencoder")
AE.compile(optimizer='adam', loss="mean_squared_error", )
TrainAE = AE.fit( x=X, y=X, epochs=100, batch_size=2**5,)

training looks good and converges smoothly, but when I look at the result the output is mainly zeros. 

f, ax = plt.subplots(figsize=(8, 8))
sns.kdeplot( X, shade=False, axis=ax)
sns.kdeplot( AE.predict(X), shade=False, axis=ax)

Blue is the original distribution, orange the autoencoded one

This seems very odd to me, because the encoding layer is as large as the input, so a trivial and loss-free solution would simple be to wire the first A neuron straight through, with an activation of 1 and same for the second neuron encoding for B. Why is this not happening? Is there any parameter I use falsely?

Upvotes: 0

Views: 53

Answers (1)

Callin
Callin

Reputation: 744

One issue is that your final layer has the relu activation, which has a minimum of 0. If you would like to predict numbers less than 0 on the final layer, you can change the activation to "linear," like this

aeOutput = Dense(2, activation='linear')(encode)

Upvotes: 1

Related Questions