CODEWITHSUNDEEP
pytorchrecurrent-neural-network
mrgloom
mrgloom

Reputation: 21612

How to reproduce RNN results on several runs?

I call same model on same input twice in a row and I don't get the same result, this model have nn.GRU layers so I suspect that it have some internal state that should be release before second run?

How to reset RNN hidden state to make it the same as if model was initially loaded?

UPDATE:

Some context:

I'm trying to run model from here:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L93

I'm calling generate:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L148

Here it's actually have some code using random generator in pytorch:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L200

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py#L110

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py#L129

I have placed (I'm running code on CPU):

torch.manual_seed(0)
torch.cuda.manual_seed_all(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)

in

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py

after all imports.

I have checked GRU weights between runs and they are the same:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L153

Also I have checked logits and sample between runs and logits are the same but sample are not, so @Andrew Naguib seems were right about random seeding, but I'm not sure where the code that fixes random seed should be placed?

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L200

UPDATE 2:

I have placed seed init inside generate and now results are consistent:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L148

Upvotes: 2

Views: 1344

Answers (2)

Anubhav Singh
Anubhav Singh

Reputation: 8699

You can use model.init_hidden() to reset the RNN hidden state.

def init_hidden(self):
     # Initialize hidden and cell states
     return Variable(torch.zeros(num_layers, batch_size, hidden_size))

So, before calling the same model on the same data next time, you can call model.init_hidden() to reset the hidden and cell states to the initial values.

This will clear out the history, in order words, the weights the model learned after running on the data first time.

Upvotes: 0

ndrwnaguib
ndrwnaguib

Reputation: 6115

I believe this may be highly related to Random Seeding. To ensure reproducible results (as stated by them) you have to seed torch as in this:

import torch
torch.manual_seed(0)

And also, the CuDNN module.

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

If you're using numpy, you could also do:

import numpy as np
np.random.seed(0)

However, they warn you:

Deterministic mode can have a performance impact, depending on your model.

A suggested script I regularly use which has been working very good to reproduce results is:

# imports
import numpy as np
import random
import torch
# ...
""" Set Random Seed """
if args.random_seed is not None:
    """Following seeding lines of code are to ensure reproducible results 
       Seeding the two pseudorandom number generators involved in PyTorch"""
    random.seed(args.random_seed)
    np.random.seed(args.random_seed)
    torch.manual_seed(args.random_seed)
    # https://pytorch.org/docs/master/notes/randomness.html#cudnn
    if not args.cpu_only:
        torch.cuda.manual_seed(args.random_seed)
        cudnn.deterministic = True
        cudnn.benchmark = False

Upvotes: 3

Related Questions