CODEWITHSUNDEEP
pythonpython-3.xtensorflow
lightbox142
lightbox142

Reputation: 152

Using tf.layer.conv proper usage - activation and activity_regularizer

In tensorflow's tf.layer.conv1d webpage https://www.tensorflow.org/api_docs/python/tf/layers/conv1d, they provide the option to set the "activation" and "activity_regularizer" for your model. I already have a model built, so it would have been nice if I could have just set activity_regularizer = tf.layer.batch_normalization(training=True) or activation=tf.nn.leaky_relu(alpha = 0.001).However, if i try to put any inputs into tf.layer.batch_normalization or tf.nn.leaky_relu they give errors saying there are arguments missing ("inputs" for former and "features" for latter).

TLDR:

          conv1 = tf.layers.conv1d(inputs=_inputs, filters=6, kernel_size=4, strides=1,
                                  padding='same', activation=tf.nn.leaky_relu, activity_regularizer = tf.layers.batch_normalization)

seems to "work" (it runs at least), but this:

          conv1 = tf.layers.conv1d(inputs=_inputs, filters=6, kernel_size=4, strides=1,
                                  padding='same', activation=tf.nn.leaky_relu(alpha = 0.001), activity_regularizer = tf.layers.batch_normalization(training = True, trainable = True))

says there are missing arguments for activation and activity_regularizer.

I may be using activation and activity_regularizer completely wrong so feel free to correct me. I am hoping there is an easy fix to this, otherwise the only option for me is to write extra lines of code to separate the activation and batch normalizations from conv1d. Although, I do not see the purpose of having activation and activity_regularizer built-in tf.layers.conv1d if I can't change their parameters.

Last point: I am particularly worried about tf.layer.batch_normalization because I am assuming it is set at the default training=False and trainable = True which should not always be the case in the block of code that "worked".

Upvotes: 1

Views: 318

Answers (1)

iga
iga

Reputation: 3633

The argument to activation should be a function that takes a tensor and returns a tensor. tf.nn.leaky_relu is such a function.

tf.nn.leaky_relu(alpha = 0.001) is not valid Python. You are not specifying the only required argument features. Python does not know how to run a function without a required argument. What you want is something like this:

def my_relu(features):
    return tf.nn.leaky_relu(features, alpha=0.001)

tf.layers.conv1d(..., activation=my_relu, ...)

The argument to activity_regularizer is a function taking your layer's activity (i.e. output) and computing a penalty for it. Usually you give high penalty for large activations. This function should output a scalar. I don't know why activity_regularizer = tf.layers.batch_normalization does not complain but it is probably doing not what you expect. Typical regularizers are l1 and l2 norms (see https://keras.io/regularizers/). Batch normalization is a layer, not a regularizer.

Upvotes: 2

Related Questions