CODEWITHSUNDEEP
tensorflow
Malo Marrec
Malo Marrec

Reputation: 609

Add L2 regularization when using high level tf.layers

Is it possible to add an L2 regularization when using the layers defined in tf.layers?

It seems to me that since tf.layers is an high level wrapper, there is no easy way to get access to the filter weights.

With tf.nn.conv2d

regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)

weights = tf.get_variable(
    name="weights",
    regularizer=regularizer
)

#Previous layers

...

#Second layer 
layer 2 = tf.nn.conv2d(
input,
weights,
[1,1,1,1],
[1,1,1,1])

#More layers
...

#Loss
loss = #some loss

reg_variables = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
reg_term = tf.contrib.layers.apply_regularization(regularizer, reg_variables)
loss += reg_term

Now what would that look like with tf.layers.conv2d?

Thanks!

Upvotes: 27

Views: 29672

Answers (4)

ywfu
ywfu

Reputation: 21

In eager execution, there are two ways.

  1. Calculate by hand with tf.add_n([tf.square(i) for i in layer.variables]) * l2_coef .
  2. Using layer.losses when the layer is created with kernel_regularizer.

As shown in the official examples: densenet_test.py

rand_input = tf.random_uniform((10, 3, 32, 32))
weight_decay = 1e-4

conv = tf.keras.layers.Conv2D(
    3, (3, 3),
    padding='same',
    use_bias=False,
    kernel_regularizer=tf.keras.regularizers.l2(weight_decay))

optimizer = tf.train.GradientDescentOptimizer(0.1)
conv(rand_input)  # Initialize the variables in the layer

def compute_true_l2(vs, wd):
  return tf.reduce_sum(tf.square(vs)) * wd

true_l2 = compute_true_l2(conv.variables, weight_decay)
keras_l2 = tf.add_n(conv.losses)
self.assertAllClose(true_l2, keras_l2)

with tf.GradientTape() as tape_true, tf.GradientTape() as tape_keras:
  loss = tf.reduce_sum(conv(rand_input))
  loss_with_true_l2 = loss + compute_true_l2(conv.variables, weight_decay)
  loss_with_keras_l2 = loss + tf.add_n(conv.losses)

true_grads = tape_true.gradient(loss_with_true_l2, conv.variables)
keras_grads = tape_keras.gradient(loss_with_keras_l2, conv.variables)
self.assertAllClose(true_grads, keras_grads)

Upvotes: 2

srcolinas
srcolinas

Reputation: 515

I see two incomplete answers, so here is the complete one:

regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)
    layer2 = tf.layers.conv2d(
        inputs,
        filters,
        kernel_size,
        kernel_regularizer=regularizer)

alternatively:

layer2 = tf.layers.conv2d(inputs, 
     filters, 
     kernel_size,                        
     kernel_regularizer= tf.contrib.layers.l2_regularizer(scale=0.1))

don't forget to add it to the final loss:

l2_loss = tf.losses.get_regularization_loss()
....
loss += l2_loss

Basically, add regularization when defining a layer and then make sure you add regularization loss to your loss.

Upvotes: 4

Robert Lacok
Robert Lacok

Reputation: 4324

You can pass them into tf.layers.conv2d as arguments:

regularizer = tf.contrib.layers.l2_regularizer(scale=0.1)
layer2 = tf.layers.conv2d(
    inputs,
    filters,
    kernel_size,
    kernel_regularizer=regularizer)

Then you should add the regularization loss to your loss like this:

l2_loss = tf.losses.get_regularization_loss()
loss += l2_loss

Edit: Thanks Zeke Arneodo, Tom and srcolinas I added, the last bit on your feedback so that the accepted answer provides the complete solution.

Upvotes: 38

Zeke Arneodo
Zeke Arneodo

Reputation: 674

Isn't the answer in your question? You can also use tf.losses.get_regularization_loss (https://www.tensorflow.org/api_docs/python/tf/losses/get_regularization_loss), which will collect all the REGULARIZATION_LOSSES.

...
layer2 = tf.layers.conv2d(input, 
     filters, 
     kernel_size,                        
     kernel_regularizer= tf.contrib.layers.l2_regularizer(scale=0.1))
...
l2_loss = tf.losses.get_regularization_loss()
loss += l2_loss

Upvotes: 16

Related Questions