CODEWITHSUNDEEP
machine-learningneural-networkpybrain
Philip O'Brien
Philip O'Brien

Reputation: 4266

Slightly differing output from Pybrain neural network despite consistent initialisation?

I am working on a feed forward network in PyBrain. To allow me to compare the effects of varying certain parameters I have initialised the network weights myself. I have done this under the assumption that if the weights are always the same then the output should always be the same. Is this assumption incorrect? Below is the code used to set up the network

n = FeedForwardNetwork()
inLayer = LinearLayer(7, name="in")
hiddenLayer = SigmoidLayer(1, name="hidden")
outLayer = LinearLayer(1, name="out")

n.addInputModule(inLayer)
n.addModule(hiddenLayer)
n.addOutputModule(outLayer)

in_to_hidden = FullConnection(inLayer, hiddenLayer, name="in-to-hidden")
hidden_to_out = FullConnection(hiddenLayer, outLayer, name="hidden-to-out")

n.addConnection(in_to_hidden)
n.addConnection(hidden_to_out)

n.sortModules()
in_to_hidden_params = [
    0.27160018, -0.30659429, 0.13443352, 0.4509613,
    0.2539234, -0.8756649, 1.25660715
    ]
hidden_to_out_params = [0.89784474]

net_params = in_to_hidden_params + hidden_to_out_params
n._setParameters(net_params)
trainer = BackpropTrainer(n, ds, learningrate=0.01, momentum=0.8)

UPDATE

It looks like even by seeding the random number generator, reproducibility is still an issue. See the GitHub issue here

Upvotes: 1

Views: 83

Answers (1)

lejlot
lejlot

Reputation: 66815

I have done this under the assumption that if the weights are always the same then the output should always be the same

The assumption is correct, but your code is not doing so. Your are training your weights, thus they do not end up being the same. Stochastic training methods often permute training samples, and this permutation leads to different results, in particular BackpropTrainer does so:

 def train(self):
        """Train the associated module for one epoch."""
        assert len(self.ds) > 0, "Dataset cannot be empty."
        self.module.resetDerivatives()
        errors = 0
        ponderation = 0.
        shuffledSequences = []
        for seq in self.ds._provideSequences():
            shuffledSequences.append(seq)
        shuffle(shuffledSequences)

If you want repeatable results - seed your random number generators.

Upvotes: 1

Related Questions