CODEWITHSUNDEEP
tensorflowkeras
bettercallsomeone
bettercallsomeone

Reputation: 33

Flag for training and test for custom layer in Keras

I want to create a custom keras layer which does something during training and something else for validation or testing.

from tensorflow import keras
K = keras.backend
from keras.layers import Layer
import tensorflow as tf

class MyCustomLayer(Layer):

    def __init__(self, ratio=0.5, **kwargs):
        self.ratio = ratio
        super(MyCustomLayer, self).__init__(**kwargs)

    @tf.function
    def call(self, x, is_training=None):

        is_training = K.learning_phase()
        tf.print("training: ", is_training)
        if is_training is 1 or is_training is True:

            xs = x * 4
            return xs
        else:
            xs = x*0
            return xs

model = Sequential()
model.add(Dense(16, input_dim=input_dim))
model.add(MyCustomLayer(0.5))
model.add(ReLU())
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(output_dim, activation='softmax', kernel_regularizer=l2(0.01)))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])


model.fit(X_train, y_train, validation_split=0.05, epochs=5)

In the output I always get:

training:  0
training:  0
training:  0
training:  0
training:  0
training:  0
training:  0
training:  0

Does anyone knows how to fix this?

Upvotes: 3

Views: 4001

Answers (2)

bettercallsomeone
bettercallsomeone

Reputation: 33

So, I just figured out what was going wrong. I was mixing two different types of classes:

from keras import Sequential
from tensorflow import keras
K = keras.backend

So, the model is using keras and I was calling the flag from tensorflow.keras. For this reason K.learning_phase() was not working as expected.

To fix it I used

from tensorflow.keras import Sequential
from tensorflow import keras
K = keras.backend

Upvotes: 0

Dr. Snoopy
Dr. Snoopy

Reputation: 56377

There are some issues and misconceptions here. First you are mixing imports between keras and tf.keras imports, you should use only one of them. Second the parameter for call is called training, not is_training.

I think the issue is that tf.print does not really print the value of the training variable as its a tensorflow symbolic variable and might change value indirectly. There are other ways to check if the layer behaves differently during inference and training, for example:

class MyCustomLayer(Layer):

def __init__(self, ratio=0.5, **kwargs):
    super(MyCustomLayer, self).__init__(**kwargs)

def call(self, inputs, training=None):

    train_x = inputs * 4
    test_x = inputs * 0

    return K.in_train_phase(train_x,
                            test_x,
                            training=training)

Using then this model:

model = Sequential()
model.add(Dense(1, input_dim=10))
model.add(MyCustomLayer(0.5))
model.compile(loss='mse', optimizer='adam')

And making an instance of a function that explictly receives the K.learning_phase() variable:

fun = K.function([model.input, K.learning_phase()], [model.output])

If you call it with Klearning_phase() set to 1 or 0 you do see different outputs:

d = np.random.random(size=(2,10))
print(fun([d, 1]))
print(fun([d, 0]))

Result:

[array([[4.1759257], [3.9988194]], dtype=float32)]
[array([[0.], [0.]], dtype=float32)]

And this indicates that the layer has differen behavior during training and inference/testing.

Upvotes: 2

Related Questions