CODEWITHSUNDEEP
pythonpython-3.xtensorflow
SUD2193
SUD2193

Reputation: 183

Difference between tf.layers.dense and tf.nn.xw_plus_b

What is the difference between tf.layers.dense and tf.nn.xw_plus_b in TF? What is the default activation used in tf.layers.dense when "activation" argument is passed as None?

Upvotes: 7

Views: 2401

Answers (1)

dm0_
dm0_

Reputation: 2156

tf.nn.xw_plus_b is a low-level operation that only computes x*W+b and requires existing variables.

tf.layers.dense is a high-level "layer" that creates variables, apply activation can set constrains and apply regularization.

According to the documentation default activation is linear (no activation).

activation: Activation function (callable). Set it to None to maintain a linear activation.

Update

In Tensorflow 1.12 Dense layer inherits keras.layers.Dense (code):

@tf_export('layers.Dense')
class Dense(keras_layers.Dense, base.Layer):

Keras implementation of this layer does the following (code):

  def call(self, inputs):
    inputs = ops.convert_to_tensor(inputs, dtype=self.dtype)
    rank = common_shapes.rank(inputs)
    if rank > 2:
      # Broadcasting is required for the inputs.
      outputs = standard_ops.tensordot(inputs, self.kernel, [[rank - 1], [0]])
      # Reshape the output back to the original ndim of the input.
      if not context.executing_eagerly():
        shape = inputs.get_shape().as_list()
        output_shape = shape[:-1] + [self.units]
        outputs.set_shape(output_shape)
    else:
      outputs = gen_math_ops.mat_mul(inputs, self.kernel)
    if self.use_bias:
      outputs = nn.bias_add(outputs, self.bias)
    if self.activation is not None:
      return self.activation(outputs)  # pylint: disable=not-callable
    return outputs

So it is not implemented using tf.nn.xw_plus_b but uses two separate operations.

To answer your question: Dense layer without activation, constraints and regularization should do the same as tf.nn.xw_plus_b.

Upvotes: 11

Related Questions