CODEWITHSUNDEEP
pytorchattention-model
Sayan Ghosh
Sayan Ghosh

Reputation: 25

Implementing self attention

I am trying to implement self attention in Pytorch. I need to calculate the following expressions.

Similarity function S (2 dimensional), P(2 dimensional), C'

S[i][j] = W1 * inp[i] + W2 * inp[j] + W3 * x1[i] * inp[j]

P[i][j] = e^(S[i][j]) / Sum for all j( e ^ (S[i]))

basically, P is a softmax function

C'[i] = Sum (for all j) P[i][j] * x1[j]

I tried the following code using for loops 

        for i in range(self.dim):
            for j in range(self.dim):
                S[i][j] = self.W1 * x1[i] + self.W2 * x1[j] + self.W3 * x1[i] * x1[j]

        for i in range(self.dim):
            for j in range(self.dim):
                P[i][j] = torch.exp(S[i][j]) / torch.sum( torch.exp(S[i]))

        # attend

        for i in range(self.dim):
            out[i] = 0
            for j in range(self.dim):
                out[i] += P[i][j] * x1[j]

Is there any faster way to implement this in Pytorch?

Upvotes: 0

Views: 1491

Answers (1)

Sayantan Das
Sayantan Das

Reputation: 349

Here is an example of Self Attention I had implemented in Dual Attention for HSI Imagery

class PAM_Module(Module):
""" Position attention module  https://github.com/junfu1115/DANet/blob/master/encoding/nn/attention.py"""
#Ref from SAGAN
def __init__(self, in_dim):
    super(PAM_Module, self).__init__()
    self.chanel_in = in_dim

    self.query_conv = Conv2d(in_channels=in_dim, out_channels=in_dim//8, kernel_size=1)
    self.key_conv = Conv2d(in_channels=in_dim, out_channels=in_dim//8, kernel_size=1)
    self.value_conv = Conv2d(in_channels=in_dim, out_channels=in_dim, kernel_size=1)

    self.gamma = Parameter(torch.zeros(1))

    self.softmax = Softmax(dim=-1)
def forward(self, x):
    """
        inputs :
            x : input feature maps( B X C X H X W)
        returns :
            out : attention value + input feature
            attention: B X (HxW) X (HxW)
    """
    m_batchsize, C, height, width = x.size()
    proj_query = self.query_conv(x).view(m_batchsize, -1, width*height).permute(0, 2, 1)
    proj_key = self.key_conv(x).view(m_batchsize, -1, width*height)
    energy = torch.bmm(proj_query, proj_key)
    attention = self.softmax(energy)
    proj_value = self.value_conv(x).view(m_batchsize, -1, width*height)

    out = torch.bmm(proj_value, attention.permute(0, 2, 1))
    out = out.view(m_batchsize, C, height, width)

    out = self.gamma*out + x
    #out = F.avg_pool2d(out, out.size()[2:4])

    return out

Upvotes: 1

Related Questions