CODEWITHSUNDEEP
pythonmachine-learningdeep-learningpytorch
Soubriquet
Soubriquet

Reputation: 3330

Regression loss functions incorrect

I'm trying a basic averaging example, but the validation and loss don't match and the network fails to converge if I increase the training time. I'm training a network with 2 hidden layers, each 500 units wide on three integers from the range [0,9] with a learning rate of 1e-1, Adam, batch size of 1, and dropout for 3000 iterations and validate every 100 iterations. If the absolute difference between the label and the hypothesis is less than a threshold, here I set the threshold to 1, I consider that correct. Could someone let me know if this is an issue with the choice of loss function, something wrong with Pytorch, or something I'm doing. Below are some plots:

val_diff = 1
acc_diff = torch.FloatTensor([val_diff]).expand(self.batch_size)

Loop 100 times to during validation:

num_correct += torch.sum(torch.abs(val_h - val_y) < acc_diff)

Append after each validation phase:

validate.append(num_correct / total_val)

Here are some examples of the (hypothesis, and labels):

[...(-0.7043088674545288, 6.0), (-0.15691305696964264, 2.6666667461395264),
 (0.2827358841896057, 3.3333332538604736)]

I tried six of the loss functions in the API that are typically used for regression:

                                                    torch.nn.L1Loss(size_average=False) enter image description here

                                                                      torch.nn.L1Loss() enter image description here

                                                     torch.nn.MSELoss(size_average=False) enter image description here

                                                                   torch.nn.MSELoss() enter image description here

                                             torch.nn.SmoothL1Loss(size_average=False) enter image description here

                                                               torch.nn.SmoothL1Loss() enter image description here

Thanks.

Network code:

class Feedforward(nn.Module):
    def __init__(self, topology):
        super(Feedforward, self).__init__()
        self.input_dim     = topology['features']
        self.num_hidden    = topology['hidden_layers']
        self.hidden_dim    = topology['hidden_dim']
        self.output_dim    = topology['output_dim']
        self.input_layer   = nn.Linear(self.input_dim, self.hidden_dim)
        self.hidden_layer  = nn.Linear(self.hidden_dim, self.hidden_dim)
        self.output_layer  = nn.Linear(self.hidden_dim, self.output_dim)
        self.dropout_layer = nn.Dropout(p=0.2)


    def forward(self, x):
        batch_size = x.size()[0]
        feat_size  = x.size()[1]
        input_size = batch_size * feat_size

        self.input_layer = nn.Linear(input_size, self.hidden_dim).cuda()
        hidden = self.input_layer(x.view(1, input_size)).clamp(min=0)

        for _ in range(self.num_hidden):
            hidden = self.dropout_layer(F.relu(self.hidden_layer(hidden)))

        output_size = batch_size * self.output_dim
        self.output_layer = nn.Linear(self.hidden_dim, output_size).cuda()
        return self.output_layer(hidden).view(output_size)

Training code:

def train(self):
    if self.cuda:
        self.network.cuda()

    dh        = DataHandler(self.data)
    # loss_fn = nn.L1Loss(size_average=False)
    # loss_fn = nn.L1Loss()
    # loss_fn = nn.SmoothL1Loss(size_average=False)
    # loss_fn = nn.SmoothL1Loss()
    # loss_fn = nn.MSELoss(size_average=False)
    loss_fn   = torch.nn.MSELoss()
    losses    = []
    validate  = []
    hypos     = []
    labels    = []
    val_size  = 100
    val_diff  = 1
    total_val = float(val_size * self.batch_size)

    for i in range(self.iterations):
        x, y = dh.get_batch(self.batch_size)
        x = self.tensor_to_Variable(x)
        y = self.tensor_to_Variable(y)

        self.optimizer.zero_grad()
        loss = loss_fn(self.network(x), y)
        loss.backward()
        self.optimizer.step()

Upvotes: 0

Views: 797

Answers (1)

Cedias
Cedias

Reputation: 904

It looks like you've misunderstood how layers in pytorch works, here are a few tips:

  • In your forward when you do nn.Linear(...) you are definining new layers instead of using those you pre-defined in your network __init__. Therefore, it cannot learn anything as weights are constantly reinitalized.

  • You shouldn't need to call .cuda() inside net.forward(...) since you've already copied the network on gpu in your train by calling self.network.cuda()

  • Ideally the net.forward(...) input should directly have the shape of the first layer so you won't have to modify it. Here you should have x.size() <=> Linear -- > (Batch_size, Features).

Your forward should look close to this:

def forward(self, x):
    x = F.relu(self.input_layer(x))
    x = F.dropout(F.relu(self.hidden_layer(x)),training=self.training)
    x = self.output_layer(x)
    return x

Upvotes: 1

Related Questions