CODEWITHSUNDEEP
tensorflow
Hasnat
Hasnat

Reputation: 674

How to implement PReLU activation in Tensorflow?

The Parametric Rectified Linear Unit (PReLU) is an interesting and widely used activation function. It seems that Tensorflow (reference link) does not provide PReLU. I know that the higher level libraries, such as Keras and TFLearn, has the implementation of it.

I would like to know how to implement PReLU in Tensorflow?

Upvotes: 10

Views: 21883

Answers (6)

BCJuan
BCJuan

Reputation: 825

You have it in Keras implemented as a layer (from version 2.3.0 onwards): tf.keras.layers.PReLU. Here is the link to the documentation page: https://www.tensorflow.org/api_docs/python/tf/keras/layers/PReLU.

It is useful if you want, for example, deploy models with TFLite and used delegates suchas NNAPI, because there are implementations specially for that NNAPI

Upvotes: 1

zsdh
zsdh

Reputation: 126

PReLU already exists in TensorLayer

http://tensorlayer.readthedocs.io/en/latest/modules/layers.html#parametric-activation-layer

Upvotes: 2

Math_Max
Math_Max

Reputation: 187

Just an addition to Hasnat's answers: (I can't comment yet.. < 50 rep)

If you want more than one different prelu layer, you should set a 'name' parameter:

def prelu(_x, name):
"""
Parametric ReLU
"""
alphas = tf.get_variable(name, _x.get_shape()[-1],
                   initializer=tf.constant_initializer(0.1),
                    dtype=tf.float32, trainable=True)
pos = tf.nn.relu(_x)
neg = alphas * (_x - abs(_x)) * 0.5

return pos + neg

Then you can give each prelu a different name, for example:

prelu(x, "alpha1")

# convolution or other

prelu(x, "alpha2")

This will resolve the error:

Variable alpha already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope?

Upvotes: 0

Szabolcs
Szabolcs

Reputation: 830

Although the solution with tf.maximum is very efficient, it can't represent a concave function. Here's a solution that can:

def prelu(_x, scope=None):
    """parametric ReLU activation"""
    with tf.variable_scope(name_or_scope=scope, default_name="prelu"):
        _alpha = tf.get_variable("prelu", shape=_x.get_shape()[-1],
                                 dtype=_x.dtype, initializer=tf.constant_initializer(0.1))
        return tf.maximum(0.0, _x) + _alpha * tf.minimum(0.0, _x)

Upvotes: 11

DboyLiao
DboyLiao

Reputation: 483

I think it's much easier to implement with tf.maximum

My implementation is as following

import tensorflow as tf

def PReLU(_x, name=None):
  if name is None:
    name = "alpha"
  _alpha = tf.get_variable(name,
                           shape=_x.get_shape(),
                           initializer=tf.constant_initializer(0.0),
                           dtype=_x.dtype)

  return tf.maximum(_alpha*_x, _x)

Upvotes: 5

Hasnat
Hasnat

Reputation: 674

The implementation of PReLU seems straight-forward based on the PreLU implementations (see: Keras, TFLearn and TensorLayer) of the higher level libraries. My code is as follows:

def parametric_relu(_x):
  alphas = tf.get_variable('alpha', _x.get_shape()[-1],
                       initializer=tf.constant_initializer(0.0),
                        dtype=tf.float32)
  pos = tf.nn.relu(_x)
  neg = alphas * (_x - abs(_x)) * 0.5

  return pos + neg

Upvotes: 21

Related Questions