CODEWITHSUNDEEP
performanceneural-networkpytorchloss-functionmse
Soni
Soni

Reputation: 1

MSE Loss is reducing with a very small amount during training pytorch

I'm trying out my first deep learning program for speech separation using ideal ratio mask. It is a BLSTM model. The mse loss obtained while training is reducing with a very small quantity. Any idea about why it is happening and how to fix it?? thank you my learning rate is 0.0003, adam optimizer, batch size 64, training dataset 3000 files and validation dataset is 340 files. I tried changing the learning rate and tried learning rate decay as well. still getting the same problem. I've been stuck here for a long time. Please do help. Thank you

Train Epoch: 1 Loss: 0.027980 
  val Epoch: 1  Loss: 0.025925
Train Epoch: 2 Loss: 0.027975 
  val Epoch: 2  Loss: 0.025961
Train Epoch: 3 Loss: 0.027973 
  val Epoch: 3  Loss: 0.025987

here is my model

import torch.nn as nn
import torch

print("Is the GPU available:", torch.cuda.is_available())  # True
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class Base_model(nn.Module):
    def __init__(self):
        super(Base_model, self).__init__()
        self.lstm = nn.LSTM(input_size=322, hidden_size=300, num_layers=4, bidirectional = True, batch_first=True)
        self.rel = nn.ReLU()
        self.fc = nn.Linear(in_features=600, out_features=161)
        self.activation = nn.Sigmoid()
        
        
    def forward(self, x):
        # print(x.size(0))

        out, _ = self.lstm(x)
        out = self.fc(out)
        out = self.activation(out)
        return out

my training code

start_epoch = 0
    if args.model_name:
        print("Load the model:", args.checkpoints_dir + args.model_name)
        checkpoint = torch.load(args.checkpoints_dir + args.model_name)
        model.load_state_dict(checkpoint["model"])
        optimizer.load_state_dict(checkpoint["optimizer"])
        start_epoch = checkpoint['epoch']
        # lr_schedule.load_state_dict(checkpoint['lr_schedule'])  # load lr_scheduler

    for epoch in range(start_epoch, args.epochs):
        model.train()  # train model
        for batch_idx, (train_X, train_mask, train_nearend_mic_magnitude, train_nearend_magnitude) in enumerate(
                train_loader):
            train_X = torch.reshape(train_X, (train_X.shape[0],train_X.shape[2],train_X.shape[1]))
            train_mask = torch.reshape(train_mask, (train_mask.shape[0],train_mask.shape[2],train_mask.shape[1]))
            train_nearend_mic_magnitude = torch.reshape(train_nearend_mic_magnitude, (train_nearend_mic_magnitude.shape[0],train_nearend_mic_magnitude.shape[2],train_nearend_mic_magnitude.shape[1]))
            train_nearend_magnitude = torch.reshape(train_nearend_magnitude, (train_nearend_magnitude.shape[0],train_nearend_magnitude.shape[2],train_nearend_magnitude.shape[1]))
            
            train_X = train_X.to(device) 
            train_mask = train_mask.to(device)  # IRM [batch_size 161, 999]
            train_nearend_mic_magnitude = train_nearend_mic_magnitude.to(device)
            train_nearend_magnitude = train_nearend_magnitude.to(device)

            # forward propagation
            optimizer.zero_grad()  # zero out the gradient
            pred_mask = model(train_X)  # [batch_size, 322, 999]--> [batch_size, 161, 999]
            train_loss = criterion(pred_mask, train_mask)
            
            # backpropagation
            train_loss.backward()  # backpropagation
            optimizer.step()  # update parameters

            # ###########    Visual printing   ############
        print('Train Epoch: {} Loss: {:.6f} '.format(epoch + 1, train_loss.item()))

Upvotes: 0

Views: 1399

Answers (1)

Shoval Eliezer
Shoval Eliezer

Reputation: 1

I would suggest u use a bigger learning rate and make a decaying learning rate:

torch.optim.lr_scheduler.StepLR

https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.StepLR.html

Upvotes: 0

Related Questions