CODEWITHSUNDEEP
pythontensorflowkerasclassification
Romeo Kienzler
Romeo Kienzler

Reputation: 3539

LOSS not changeing in very simple KERAS binary classifier

I'm trying to get a very (over) simplified Keras binary classifier neural network running without success. The LOSS just stays constant. I've played around with Optimizers (SGD, Adam, RMSProp), Learningrates, Weight-Initializations, Batch Size and input data normalization so far.

Nothing changes at all. Am I doing something fundamentally wrong? Here is the code:

from tensorflow import keras
from keras import Sequential
from keras.layers import Dense
from keras.optimizers import SGD

data = np.array(
    [
        [100,35,35,12,0],
        [101,46,35,21,0],
        [130,56,46,3412,1],
        [131,58,48,3542,1]
    ]
)

x = data[:,1:-1]
y_target = data[:,-1]

x = x / np.linalg.norm(x)

model = Sequential()
model.add(Dense(3, input_shape=(3,), activation='softmax', kernel_initializer='lecun_normal',
                bias_initializer='lecun_normal'))
model.add(Dense(1, activation='softmax', kernel_initializer='lecun_normal',
                bias_initializer='lecun_normal'))

model.compile(optimizer=SGD(learning_rate=0.1),
              loss='binary_crossentropy',
              metrics=['accuracy'])

model.fit(x, y_target, batch_size=2, epochs=10,
          verbose=1)

Upvotes: 0

Views: 55

Answers (2)

Romeo Kienzler
Romeo Kienzler

Reputation: 3539

This is the working solution based on the feedback I got

from tensorflow import keras
from keras import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
from keras.utils import to_categorical

data = np.array(
    [
        [100,35,35,12,0],
        [101,46,35,21,0],
        [130,56,46,3412,1],
        [131,58,48,3542,1]
    ]
)

x = data[:,1:-1]
y_target = data[:,-1]

x = x / np.linalg.norm(x)

model = Sequential()
model.add(Dense(3, input_shape=(3,), activation='sigmoid'))
model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer=SGD(learning_rate=0.1),
              loss='binary_crossentropy',
              metrics=['accuracy'])

model.fit(x, y_target, epochs=1000,
          verbose=1)

Upvotes: 0

Geeocode
Geeocode

Reputation: 5797

Softmax definition is: 

exp(a) / sum(exp(a)

so when you use with a single neuron you will get:

exp(a) / exp(a) = 1

That is why your classifier doesn't work with a single neuron.

You can use sigmoid instead in this special case: 

exp(a) / (exp(a) + 1)

Furthermore sigmoid function is for two class classifiers. Softmax is an extension of sigmoid for multiclass classifers.

For the first layer you should use relu or sigmoid function instead of softmax.

Upvotes: 1

Related Questions