CODEWITHSUNDEEP
python-3.xpytorch
user9913405
user9913405

Reputation:

Pytorch : significance of grad_h_relu.clone() in Backprop

I am learning Pytorch, while I'm looking to the tutorial on their site I can't understand the significance of grad_h = grad_h_relu.clone().

h = x.mm(w1)
h_relu = h.clamp(min=0)
y_pred = h_relu.mm(w2)

# Compute and print loss
loss = (y_pred - y).pow(2).sum().item()
print(t, loss)

# Backprop to compute gradients of w1 and w2 with respect to loss
grad_y_pred = 2.0 * (y_pred - y)
grad_w2 = h_relu.t().mm(grad_y_pred)
grad_h_relu = grad_y_pred.mm(w2.t())


grad_h = grad_h_relu.clone() # what is the signifigance of this line?


grad_h[h < 0] = 0
grad_w1 = x.t().mm(grad_h)

# Update weights using gradient descent
w1 -= learning_rate * grad_w1
w2 -= learning_rate * grad_w2

Upvotes: 1

Views: 123

Answers (1)

Avijit Dasgupta
Avijit Dasgupta

Reputation: 2065

grad_h = grad_h_relu.clone()

It means that you are making a copy of the gradient of relu such that it does not share the memory with the original grad_h_relu. Then you perform some operation on it. As they are stored in two different locations, changing the value of grad_h by the following operation will not affect the grad_h_relu.

grad_h[h < 0] = 0
grad_w1 = x.t().mm(grad_h)

This grad_w1 is needed to update your parameters of your network.

Upvotes: 1

Related Questions