CODEWITHSUNDEEP
pythontensorflowkerasdeep-learning
WhaleShark
WhaleShark

Reputation: 85

Keras weights of first layer didn't change

I'm very new to Keras and I'm writing a custom layer which implements Gaussian function [exp(-(w*x-mean)^2/sigma^2) where W, mean, sigma are all randomly generated].
Below is code for the custom layer: 

class Gaussian(Layer):
    def __init__(self,**kwargs):
        super(Gaussian, self).__init__(**kwargs)

    def build(self, input_shape):
        # Create trainable weights for this layer.
        self.W_init = np.random.rand(1,input_shape[1])
        self.W = K.variable(self.W_init, name="W")

        # Create trainable means for this layer.
        self.mean_init = np.random.rand(1,input_shape[1])
        self.mean = K.variable(self.mean_init, name="mean")

        # Create trainable sigmas for this layer.
        self.sigma_init = np.random.rand(1,input_shape[1])
        self.sigma = K.variable(self.sigma_init, name="sigma")

        self.trainable_weights = [self.mean, self.sigma]
        super(Gaussian, self).build(input_shape)  # Be sure to call this somewhere!

    def call(self, x):
        result = tf.multiply(x, self.W)
        result = tf.subtract(x, self.mean)
        result = tf.multiply(tf.square(result),-1)
        result = tf.divide(result, tf.square(self.sigma))
        return result

    def compute_output_shape(self, input_shape):
        return input_shape

After putting it as the first layer in a Keras mnist tutorial(just wanted to make sure it runs without producing errors, didn't care for accuracy) and training the model, it appeared that the loss stopped decreasing after around 4 epochs and only the numbers of "mean" and "sigma" changed after training while the numbers of "W" remains the same. However, this doesn't happen if I put it as the second layer.

I ran the Keras mnist tutorial again without the custom layer and found out that the weights of the first layer didn't change either.

Is not updating the weights of first layer(more specifically the very first parameter) a Keras thing or am I missing something? Can I force it to update?
Thank you!

Upvotes: 3

Views: 1085

Answers (1)

Dr. Snoopy
Dr. Snoopy

Reputation: 56407

You are not implementing your layer correctly, Keras is not aware of your weights, that means they are not being trained by gradient descent. Take a look at this example:

from keras import backend as K
from keras.engine.topology import Layer
import numpy as np

class MyLayer(Layer):

    def __init__(self, output_dim, **kwargs):
        self.output_dim = output_dim
        super(MyLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        # Create a trainable weight variable for this layer.
        self.kernel = self.add_weight(name='kernel', 
                                      shape=(input_shape[1], self.output_dim),
                                      initializer='uniform',
                                      trainable=True)
        super(MyLayer, self).build(input_shape)  # Be sure to call this at the end

    def call(self, x):
        return K.dot(x, self.kernel)

    def compute_output_shape(self, input_shape):
        return (input_shape[0], self.output_dim)

Here you have to use add_weight to obtain a trainable weight, not just use K.variable as you are currently doing. This way your weights will be registered with Keras and they will be trained properly. You should do this for all trainable parameters in your layer.

Upvotes: 2

Related Questions