CODEWITHSUNDEEP
pythonneural-networkpytorch
Gabrosciurti
Gabrosciurti

Reputation: 45

Create a new model in pytorch with custom initial value for the weights

I'm new to pytorch and I want to understand how can I set the initial weight for the first hidden layer of my network. I explain a little better: my network is a very simple one layer MLP with 784 inputs values and 10 output values

 class Classifier(nn.Module):
        def __init__(self):
          super().__init__()
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 10)
        # Dropout module with 0.2 drop probability
        self.dropout = nn.Dropout(p=0.2)

    def forward(self, x):
        # make sure input tensor is flattened
        # x = x.view(x.shape[0], -1)

        # Now with dropout
        x = self.dropout(F.relu(self.fc1(x)))

        # output so no dropout here
        x = F.log_softmax(self.fc2(x), dim=1)

        return x

and I have, for now, a numpy matrix of shape(128, 784) witch contain the values that I want for the weight in fc1. How can I initialise the weights of the first layer with the value contained in the matrix?

Searching online in other answers I find out that I have to define the init function for the weight, e.g. 

def weights_init(m):
    classname = m.__class__.__name__

    if classname.find('Conv2d') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

but i can not understand the code

Upvotes: 3

Views: 4807

Answers (1)

Dishin H Goyani
Dishin H Goyani

Reputation: 7693

You can use simply torch.nn.Parameter() to assign a custom weight for the layer of your network.

As in your case - 

model.fc1.weight = torch.nn.Parameter(custom_weight)

torch.nn.Parameter: A kind of Tensor that is to be considered a module parameter.

For Example:

# Classifier model
model = Classifier()

# your custom weight, here taking randam
custom_weight = torch.rand(model.fc1.weight.shape)
custom_weight.shape
torch.Size([128, 784])

# before assign custom weight
print(model.fc1.weight)
Parameter containing:
tensor([[ 1.6920e-02,  4.6515e-03, -1.0214e-02,  ..., -7.6517e-03,
          2.3892e-02, -8.8965e-03],
        ...,
        [-2.3137e-02,  5.8483e-03,  4.4392e-03,  ..., -1.6159e-02,
          7.9369e-03, -7.7326e-03]])

# assign custom weight to first layer
model.fc1.weight = torch.nn.Parameter(custom_weight)

# after assign custom weight
model.fc1.weight
Parameter containing:
tensor([[ 0.1724,  0.7513,  0.8454,  ...,  0.8780,  0.5330,  0.5847],
        [ 0.8500,  0.7687,  0.3371,  ...,  0.7464,  0.1503,  0.7720],
        [ 0.8514,  0.6530,  0.6261,  ...,  0.7867,  0.9312,  0.3890],
        ...,
        [ 0.5426,  0.7655,  0.1191,  ...,  0.4343,  0.2500,  0.6207],
        [ 0.2310,  0.4260,  0.4138,  ...,  0.1168,  0.5946,  0.2505],
        [ 0.4220,  0.5500,  0.6282,  ...,  0.5921,  0.7953,  0.9997]])

Upvotes: 5

Related Questions