CODEWITHSUNDEEP
kerasrecurrent-neural-network
Mbaps
Mbaps

Reputation: 143

Gradient clipping in keras

I have a fully implemented LSTM RNN using Keras, and I want to use gradient clipping with the gradient norm limited to 5 (I'm trying to reproduce a research paper). I'm quite a beginner with regards to implementing Neural Networks, how would I implement this ?

Is it just (I'm using rmsprop optimizer):

 sgd = optimizers.rmsprop(lr=0.01, clipnorm=5)

 model.compile(optimizer=sgd,
                  loss='categorical_crossentropy',
                  metrics=['accuracy'])

Upvotes: 6

Views: 12098

Answers (1)

ptyshevs
ptyshevs

Reputation: 1672

According to the official documentation, any optimizer can have optional arguments clipnorm and clipvalue. If clipnorm provided, gradient will be clipped whenever gradient norm exceeds the threshold.

Upvotes: 8

Related Questions