CODEWITHSUNDEEP
pythontensorflowkeras
GoTN
GoTN

Reputation: 135

Incorporate side conditions into Keras neural network

I want to train my neural network (in Keras) with an additional condition on the output elements.

An example:

Without the condition, the task is straightforward. Note: The condition is not necessarily the vector norm of y_pred.

How can I implement the additional condition/restriction in a Keras (or maybe Tensorflow) model?

Upvotes: 0

Views: 204

Answers (1)

lhk
lhk

Reputation: 30196

In principle, tensorflow (and keras) don't allow you to add hard constraints to your model.

You have to convert your invarient (norm <= 1) to a penalty function, which is added to the loss. This could look like this:

y_norm = tf.norm(y_pred)
norm_loss = tf.where(y_norm > 1, y_norm, 0)
total_loss = mse + norm_loss

Look at the docs of where. If your prediction has a norm bigger than one, backpropagation tries to minimize the norm. If it is less than or equal, this part of the loss is simply 0. No gradient is produced.

But this can be very hard to optimize. Your predictions could oscillate around a norm of 1. It is also possible to add a factor: total_loss = mse + 1000* norm_loss. Be very careful with this, it makes optimization even harder. In the example above, the norm above one contributes linearly to the loss. This is called l1-regularization. You could also square it, which would become l2-regularization.

In your specific case, you could get creative. Why not normalize your predictions and the targets to one (just a suggestion, might be a bad idea)?

loss = mse(y_pred / tf.norm(y_pred), y_target / np.linalg.norm(y_target)

Upvotes: 1

Related Questions