CODEWITHSUNDEEP
pytorch
Saeed
Saeed

Reputation: 718

Why torch.dot(a,b) makes requires_grad=False

I have some losses in a loop storing them in a tensor loss. Now I want to multiply a weight tensor to the loss tensor to have final loss, but after torch.dot(), the result scalar, ll_new, has requires_grad=False. The following is my code.

loss_vector = torch.FloatTensor(total_loss_q)
w_norm = F.softmax(loss_vector, dim=0)
ll_new = torch.dot(loss_vector,w_norm)

How can I have requires_grad=False for the ll_new after doing the above?

Upvotes: 0

Views: 989

Answers (2)

Alexander Pivovarov
Alexander Pivovarov

Reputation: 4990

The issue most likely lies within this part:

I have some losses in a loop storing them in a tensor loss

You are most likely losing requires_grad somewhere in the process before torch.dot. E.g. if you use something like .item() on individual losses when constructing total_loss_q tensor.

What type is your total_loss_q? If it is a list of integers then there is no way your gradients will propagate through that. You need to construct total_loss_q in such a way that it is a tensor which knows how each individual loss was constructed (i.e. can propagate gradients to your trainable weights).

Upvotes: 0

Wasi Ahmad
Wasi Ahmad

Reputation: 37761

I think the issue is in the line: loss_vector = torch.FloatTensor(total_loss_q) as requires_grad for loss_vector is False (default value). So, you should do:

loss_vector = torch.FloatTensor(total_loss_q, requires_grad=True)

Upvotes: 2

Related Questions