CODEWITHSUNDEEP
keras
shubham gupta
shubham gupta

Reputation: 33

Thresholded Linear Layer based on Maximum and Minimum of output values

I am working on a neural network architecture which has a linear layer, and I need the output of the layer to be same as input if it is above a certain threshold, i.e

a(x) = x if x >= threshold      else a(x) = 0 if x < threshold

And the linear layer is as follows:

t = Dense(100)

Therefore, I am using the ThresholdedReLU layer after the Dense layer in keras. The threshold is such that it depends on the maximum and minimum of the output values of the Dense layer:

threshold = delta*min{s} + (1-delta)*max{s}
where min{s} is the minimum of the 100 output values of the Dense layer
and   max{s} is the maximum of the 100 output values of the Dense layer
and   delta is a value between [0,1]

Is there a way I could obtain the maximum and minimum values, calculate the threshold after each epoch and batch update, and hence obtain the thresholded output

Upvotes: 3

Views: 686

Answers (1)

rvinas
rvinas

Reputation: 11895

You could define a Lambda layer and use backend functions within it. Here's how I would do it:

from keras.layers import Dense, Lambda
from keras.models import Sequential
import keras.backend as K
import numpy as np


def thresholded_relu(x, delta):
    threshold = delta * K.min(x, axis=-1) + (1 - delta) * K.max(x, axis=-1)
    return K.cast((x > threshold[:, None]), dtype=K.dtype(x)) * x


delta = 0.5
model = Sequential()
# model.add(Dense(100, input_shape=(100,)))
model.add(Lambda(lambda x: thresholded_relu(x, delta), input_shape=(100,)))
model.compile('sgd', 'mse')

x = np.arange(0, 100, 1)[None, :]
pred = model.predict(x)
for y, p in zip(x[0], pred[0]):
    print('Input: {}. Pred: {}'.format(y, p))

Upvotes: 2

Related Questions