CODEWITHSUNDEEP
pythonpytorch
gis_grad_student
gis_grad_student

Reputation: 184

Dot product along a dimension

I have two tensors of shape [B, 3 , 240, 320] where B represents the batch size 3 represents the channels, 240 the height(H), 320 the width(W).

I need to find the dot product along the channels dimension(3 channels) thus the resulting tensor would be of shape [B, 1 , 240, 320]. My tensors have float32 elements in gpu(cuda to backprop).

Can you all please suggest how I can do that?

Thanks!

More clarification:

Let's say we have B=10, H=100, W=200. So from the above would be common for both the first and seconds tensors. If we keep B, H, W constant we get a 1D vector as the resultant tensor(with 3 elements). I need to take the dot product of these two vectors. Thus the resultant tensor is of dimension [B, 1, 240, 320]

Upvotes: 3

Views: 1905

Answers (2)

Fırat Kıyak
Fırat Kıyak

Reputation: 469

@yutasrobot's answer is fine, but sometimes you don't want to compute the intermediate multiplication explicitly. If the broadcasted shape is too large it can cause too much memory consumption. It also hurts the performance probably because it ruins the cache gains. Take a look at this answer for more on caches.

The alternative solution is to use matrix multiplication or einsum, which do not copy the products into the RAM and then fetch it again.

>>> A = torch.randn(1000, 500, 200)
>>> B = torch.randn(1000, 500, 1)
>>> torch.allclose((A * B).sum(dim=-1), torch.einsum('abc,abc->ab', A, B), atol = 1e-4)
True

>>> %timeit (A * B).sum(dim=-1)
181 ms ± 856 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

>>> %timeit torch.einsum('abc,abc->ab', A, B)
49.4 ms ± 494 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Bonus:

I noticed that the following outputs False:

>>> torch.allclose((A * B).sum(dim=-1), torch.einsum('abc,abc->ab', A, B))
False

After checking whether I did something wrong, I realized this is not a problem of einsum but a problem of the latter. Not computing the intermediary products also helps with numerical accuracy. Storing the multiplications in another tensor creates many floating point errors whereas summing the products on the fly can obtain a better accuracy. This can be shown with the following example:

>>> C = torch.randn(1000000)
>>> D = torch.randn(1000000)

>>> (C * D).sum()
tensor(191.5947)

>>> torch.dot(C, D)
tensor(191.5979)

>>> torch.einsum('i,i->', C, D)
tensor(191.5979)

Upvotes: 0

yutasrobot
yutasrobot

Reputation: 2496

Dot product is the summation of multiplication of values in two vectors:

So I am guessing you want to multiply all values along the channel dimension and need to find the summation of the result, please correct me if my understanding is wrong.

import torch

t1 = torch.rand(10, 3, 240, 320)
t2 = torch.rand(10, 3, 240, 320)

# Multiply two tensors and sum along the channel dimension
multp  = t1 * t2 
summed = multp.sum(dim = 1, keepdim = True)

print(summed.shape)  # torch.Size([10, 1, 240, 320])

Upvotes: 2

Related Questions