CODEWITHSUNDEEP
pythontensorflowneural-networkdeep-learning
J. Ali
J. Ali

Reputation: 33

Is there a way to add constraints to a neural network output but still with softmax activation function?

I am not a deep learning geek, i am learning to do this for my homework. How can I make my neural network output a list of positive floats that sum to 1 but at the same time each element of the list is smaller than a treshold (0.4 for example)?

I tried to add some hidden layers before the output layer but that didn't improve the results.
here is where i started from:

def build_net(inputs, predictor,scope,trainable):
    with tf.name_scope(scope):
        if predictor == 'CNN':
            L=int(inputs.shape[2])
            N = int(inputs.shape[3])
            conv1_W = tf.Variable(tf.truncated_normal([1,L,N,32], stddev=0.15), trainable=trainable)
            layer = tf.nn.conv2d(inputs, filter=conv1_W, padding='VALID', strides=[1, 1, 1, 1])
            norm1 = tf.layers.batch_normalization(layer)
            x = tf.nn.relu(norm1)

            conv3_W = tf.Variable(tf.random_normal([1, 1, 32, 1], stddev=0.15), trainable=trainable)
            conv3 = tf.nn.conv2d(x, filter=conv3_W, strides=[1, 1, 1, 1], padding='VALID')
            norm3 = tf.layers.batch_normalization(conv3)
            net = tf.nn.relu(norm3)

            net=tf.layers.flatten(net)

            return net




x=build_net(inputs,predictor,scope,trainable=trainable)
y=tf.placeholder(tf.float32,shape=[None]+[self.M])

network = tf.add(x,y)
w_init=tf.random_uniform_initializer(-0.0005,0.0005) 
outputs=tf.layers.dense(network,self.M,activation=tf.nn.softmax,kernel_initializer=w_init)

I am expecting that outputs would still sum to 1, but each element of it is smaller than a specific treshold i set.

Thank you so much in advance for your precious help guys.

Upvotes: 3

Views: 1829

Answers (1)

Dr. Snoopy
Dr. Snoopy

Reputation: 56357

What you want to do is add a penalty in case any of the outputs is larger than some specified thresh, you can do this with the max function:

thresh = 0.4
strength = 10.0
reg_output = strength * tf.reduce_sum(tf.math.maximum(0.0, outputs - thresh), axis=-1)

Then you need to add reg_output to your loss so its optimized with the rest of the loss. strength is a tunable parameter that defines how strong is the penalty for going over the threshold, and you have to tune it to your needs.

This penalty works by summing max(0, output - thresh) over the last dimension, which activates the penalty if output is bigger than thresh. If its smaller, the penalty is zero and does nothing.

Upvotes: 4

Related Questions