CODEWITHSUNDEEP
pythonnumpypytorch
Amir
Amir

Reputation: 11096

How to use a numpy function as the loss function in PyTorch and avoid getting errors during run time?

For my task, I do not need to compute gradients. I am simply replacing nn.L1Loss with a numpy function (corrcoef) in my loss evaluation but I get the following error:

RuntimeError: Can’t call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

I couldn’t figure out how exactly I should detach the graph (I tried torch.Tensor.detach(np.corrcoef(x, y)) but I still get the same error. I eventually wrapped everything using with torch.no_grad as follow:

with torch.no_grad():
    predFeats = self.forward(x)
    targetFeats = self.forward(target)
    loss = torch.from_numpy(np.corrcoef(predFeats.cpu().numpy().astype(np.float32), targetFeats.cpu().numpy().astype(np.float32))[1][1])

But this time I get the following error:

TypeError: expected np.ndarray (got numpy.float64)

I wonder, what am I doing wrong?

Upvotes: 2

Views: 3246

Answers (1)

Vaisakh
Vaisakh

Reputation: 331

TL;DR

with torch.no_grad():
    predFeats = self(x)
    targetFeats = self(target)
    loss = torch.tensor(np.corrcoef(predFeats.cpu().numpy(),
                                    targetFeats.cpu().numpy())[1][1]).float()

You would avoid the first RuntimeError by detaching the tensors (predFeats and targetFeats) from the computational graph. i.e. Getting a copy of the tensor data without the gradients and the gradient function (grad_fn).

So, instead of

torch.Tensor.detach(np.corrcoef(x.numpy(), y.numpy())) # Detaches a newly created tensor!
# x and y still may have gradients. Hence the first error.

which does nothing, do

# Detaches x and y properly
torch.Tensor(np.corrcoef(x.detach().numpy(), y.detach().numpy()))

But let's not bother with all the detachments.

Like you rightfully fixed, it, let's disable the gradients.

torch.no_grad()

Now, compute the features.

predFeats = self(x) # No need for the explicit .forward() call
targetFeats = self(target)

I found it helpful to break your last line up.

loss = np.corrcoef(predFeats.numpy(), targetFeats.numpy()) # We don't need to detach

# Notice that we don't need to cast the arguments to fp32
# since the `corrcoef` casts them to fp64 anyway.

print(loss.shape, loss.dtype) # A 2-dimensional fp64 matrix

loss = loss[1][1]
print(type(loss)) # Output: numpy.float64
# Loss now just a simple fp64 number

And that is the problem!

Because, when we do

loss = torch.from_numpy(loss)

we're passing in a number (numpy.float64) while it expects a numpy tensor (np.ndarray).

If you're using PyTorch 0.4 or up, there's inbuilt support for scalars.

Simply replace the from_numpy() method with the universal tensor() creation method.

loss = torch.tensor(loss)

P.S. You might also want to look at setting rowvar=False in corrcoef since the rows in PyTorch tensors usually represent the observations.

Upvotes: 3

Related Questions