How to implement this equation in pytorch?

Question

I'm trying to implement GNNs from a research paper and I have to code the following equations to get some sort of relatedness scores sj.

Since I am new to pytorch, I'm having some difficulties implementing equations.

Following are the set of equations that I want to code:

I have the following inputs h(t) = input_a = hidden[:,0:8,:]

hc(t) = input_b = hidden[:,8:13,:] The dimensions of 'hidden' is (1000, 13, 128) #(Batch size, inputs(context-0:8, and candidates 8:13), embedding dimension)

Also, the subscripts i in h(t) belong to 0:8, and j in hc(t) belong to 8:13. So, hi(t) would be hidden[:,i,:].

To provide some context,

Could someone please help me code the equations in the first picture ? I can even implement the function g() by myself, but I'm confused about the first two. Could someone help ?

Answer 1

All the operations you could do to pytorch tensors are documented over here: https://pytorch.org/docs/stable/torch.html I suggest command + F to search for the operation you need.

For the 1st equations you gave:

• You can find torch.tanh() as shown here: https://pytorch.org/docs/stable/generated/torch.tanh.html#torch.tanh
• You can find torch.transpose() over here https://pytorch.org/docs/stable/generated/torch.transpose.html#torch.transpose (Also has 2 aliases torch.swapaxes and torch.swapdims that you can find in the documentation I linked above)

For the second equation

• You can find the exponential with torch.exp() as shown here https://pytorch.org/docs/stable/generated/torch.exp.html#torch.exp
• The summation can be coded with a for loop or clever indexing in pytorch.

There are many operations you can do to a torch.tensor object that are pretty useful for papers. As a rule of thumb, when implementing papers just use this pytorch doc as a way to find out many of those operations are.

Additionally if you are less comfortable with pytorch, another suggestion can be to do the equations in numpy then convert that value to a tensor.

Sarthak