CODEWITHSUNDEEP
parametersdeep-learningpytorchdata-loss
santa
santa

Reputation: 1

Loss Analysis of Deep Learning

I'm new to deep learning, and I have built a graph convolution network. I used 5 fold cross-validations. After plotting the mean train_loss (blue) and validate_loss (orange) together, I got this baby.

MSE loss

As you can see, from the curvilinear trend of validate_loss, it seems that my networks learn few things. (I guess data? GCN frameworks? learning rate?)

Could you guys specifically help me to figure out the bugs?

I would appreciate that! If you don't get my point, please let me know.

class Scorer(nn.Module):
    """
    Three conv_layers and two fc_layers with Dropout
    """
    def __init__(self):
        super(Scorer, self).__init__()
        self.conv_layer1 = GraphConvNet(5, 64)
        self.conv_layer2 = GraphConvNet(64, 128)
        self.conv_layer3 = GraphConvNet(128, 256) # (I have tried delete conv_layer3)
        self.fc_layer1 = nn.Linear(256, 128)
        self.drop_layer1 = nn.Dropout(0.5)
        self.fc_layer2 = nn.Linear(128, 64)
        self.drop_layer2 = nn.Dropout(0.5)
        self.out_layer = nn.Linear(64, 1)


    def forward(self, NormLap, feat):
        h = self.conv_layer1(NormLap, feat)
        h = F.leaky_relu(h)
        h = self.conv_layer2(NormLap, h)
        h = F.leaky_relu(h)
        h = self.conv_layer3(NormLap, h)
        h = F.leaky_relu(h)
        h = self.fc_layer1(h)
        h = self.drop_layer1(h)
        h = F.leaky_relu(h)
        h = self.fc_layer2(h)
        h = self.drop_layer2(h)
        h = F.leaky_relu(h)
        h = self.out_layer(h)
        h = F.leaky_relu(h)

Below is my networks and parameters:

    # parameter setting
    learning_rate = 0.001 # (I have tried 1e-1, 1e-2)
    weight_decay = 1e-3  # (I have tried 1e-4)
    epochs = 500
    batch_size = 50  # (I have tried 30)

    model = Scorer()
    loss_func = nn.MSELoss()
    optimizer = th.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay)

Upvotes: 0

Views: 103

Answers (1)

Alex I
Alex I

Reputation: 20287

This is pretty much what train and validation loss is supposed to do. Loss goes down over time; that's what an optimizer is trying to do. train_loss keeps going down after valid_loss levels out or plateaus, indicating the model starts overfitting after epoch ~100 or so. Whether MSE 0.3 is good or bad for your application depends entirely on the application, but yes, the optimizer is optimizing just fine.

Please have a look at this resource for how to interpret loss curves: https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/

"from the curvilinear trend of validate_loss, it seems that my networks learn few things" - It would help if you explain (in a lot of detail :) WHY you think that, in order to get better answers. What do you expect to see other than what you see? I look at the same graph and it seems to me your network is learning to model the data and predict whatever you're trying to predict.

Upvotes: 2

Related Questions