CODEWITHSUNDEEP
tensorflowkerasconstraints
SarahJuliet1510
SarahJuliet1510

Reputation: 120

Is it possible to enforce mathematical constraints between tensorflow neural network output nodes?

Is it possible to enforce mathematical constraints between tensorflow neural network output nodes in the last layer?

For example, monotonicity between nodes, such as output node 1 being larger than node 2, which in turn is larger than node 3, and so forth.

Upvotes: 3

Views: 778

Answers (2)

Jirayu Kaewprateep
Jirayu Kaewprateep

Reputation: 766

There are many ways one is neurons learning and the second is mathematics Fn or both by scores weights or rewards or learning from other neurons.

This is one way I teach the networks to lean, you may study from the online sample.

X = tf.compat.v1.placeholder(tf.float32, shape=( (10, 88, 80, 4)))
y = tf.compat.v1.placeholder(tf.float32, shape=(1, 1))
X_action = tf.compat.v1.get_variable('X_action', dtype = tf.float32, initializer = tf.random.normal((1, 1))) # X_var
in_training_mode = tf.compat.v1.get_variable('in_training_mode', dtype = tf.float32, initializer = tf.random.normal((1, 1))) # X_var

loss = tf.reduce_mean(input_tensor=tf.square((X * y) - (X * X_action))) *
optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)

This is another way me for the similar tasks with simple configuration just select the correct optimizer and loss Fn that performs faster learning, the previous sample is just Error roots mean sequare.

optimizer = tf.keras.optimizers.Nadam(
    learning_rate=0.00001, beta_1=0.9, beta_2=0.999, epsilon=1e-07,
    name='Nadam'
)
lossfn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer=optimizer, loss=lossfn, metrics=['accuracy'])current version.

Upvotes: -1

xdurch0
xdurch0

Reputation: 10474

In general -- not really, not directly, at least. Keras layers support arguments for constraints on the weights, and you may be able to translate a desired output constraint into a weight constraint instead -- but otherwise, you will need to think about how to set up the structure of your network such that the constraints are fulfilled.

Here is a sketch for how a monotonicity constraint might be possible. Actually including this in a model likely requires creating a custom Layer subclass or perhaps using the functional API.

First, let's create some dummy data. This could be the output of a standard Dense layer (4 is batch size, 5 the number of outputs).

raw_outputs = tf.random.normal([4, 5])
>>> <tf.Tensor: shape=(4, 5), dtype=float32, numpy=
array([[ 0.3989258 , -1.7693167 ,  0.13419539,  1.1059834 ,  0.3271042 ],
       [ 0.6493515 , -1.4397207 ,  0.05153034, -0.2730962 , -1.1569825 ],
       [-1.3043666 ,  0.20206456, -0.3841469 ,  1.8338723 ,  1.2728293 ],
       [-0.3725195 ,  1.1708363 , -0.01634515, -0.01382025,  1.2707714 ]],
      dtype=float32)>

Next, make all outputs be positive using softplus. Think of this as the output activation function. Any function that returns values >= 0 will do. For example, you could use tf.exp but the exponential growth might lead to numerical issues. I would not recommend relu since the hard 0s prevent gradients from flowing -- usually a bad idea in the output layer.

positive_outputs = tf.nn.softplus(raw_outputs)
>>> <tf.Tensor: shape=(4, 5), dtype=float32, numpy=
array([[0.9123723 , 0.15738781, 0.7624942 , 1.3918277 , 0.8700147 ],
       [1.0696293 , 0.21268418, 0.71924424, 0.56589293, 0.2734058 ],
       [0.24007489, 0.7992745 , 0.5194075 , 1.9821143 , 1.5197192 ],
       [0.5241344 , 1.4409455 , 0.685008  , 0.68626094, 1.5181118 ]],
      dtype=float32)>

Finally, use cumsum to add up the values:

constrained = tf.cumsum(positive_outputs, reverse=True, axis=-1)

>>> <tf.Tensor: shape=(4, 5), dtype=float32, numpy=
array([[4.0940967, 3.1817245, 3.0243368, 2.2618425, 0.8700147],
       [2.8408566, 1.7712271, 1.558543 , 0.8392987, 0.2734058],
       [5.0605907, 4.8205156, 4.021241 , 3.5018334, 1.5197192],
       [4.8544607, 4.3303266, 2.889381 , 2.204373 , 1.5181118]],
      dtype=float32)>

As we can see, the outputs for each batch element are monotonically decreasing! This is because each of our original outputs (positive_outputs) basically just encodes how much is added at each unit, and because we forced them to be positive, the numbers can only get larger (or smaller in this case because of reverse=True in cumsum).

Upvotes: 3

Related Questions