CODEWITHSUNDEEP
pythonneural-networkpytorch
Sivaramakrishnan
Sivaramakrishnan

Reputation: 11

Dynamic addition of hidden units in pytorch

I am trying to add hidden units to a 3-layered neural network(input, hidden,output) dynamically as I train it. I want to keep the weights of trained part of the network as I add new hidden units.This is my code,

class my_network(torch.nn.Module):
    def __init__(self,input_dim,hidden_dim,output_dim):
        super(my_network,self).__init__()
        self.I = input_dim
        self.H = hidden_dim
        self.O = output_dim
        self.layer1 = torch.nn.Linear(input_dim,hidden_dim)
        self.layer2 = torch.nn.Linear(hidden_dim,output_dim)

    def add_neurons(self,no_of_neurons,flag):
        if flag == 1:
            weights = [self.layer1.weight.data,self.layer2.weight.data]
            self.layer1 = torch.nn.Linear(self.I,self.H+no_of_neurons)
            self.layer2 = torch.nn.Linear(self.H+no_of_neurons,self.O)
            self.layer1.weight.data[0:-no_of_neurons,:] = weights[0]
            self.layer2.weight.data[:,0:-no_of_neurons] = weights[1]
            self.H = self.H + no_of_neurons
        return self.layer1.weight.shape[0]

    def forward(self,x):
        temp = self.layer1(x)
        out = self.layer2(temp)
        return out

I have noticed that once I call “add_neurons” method, the weights stop updating(while gradients are generated). Any help would be much appreciated.

Upvotes: 1

Views: 932

Answers (1)

Egor Lakomkin
Egor Lakomkin

Reputation: 1434

The optimizer might not be informed about new parameters that you added to your model. The easiest probably would be to recreate optimizer object with the updated list of your model parameters.

Upvotes: 1

Related Questions