Shaki
Shaki

Reputation: 29

How to use pytorch to perform gradient descent, and print out the true minimum?

I'm trying to use a gradient method to find a minimizer of a function f, where x(k+1)=x(k)-α▽f(x), and α=0.1. The function f is f(x) = 2*(x^3+1)^(-1/2), x(0)=1 Here is the pytorch sample

import torch
import torch.nn as nn
from torch import optim

alpha = 0.1

class MyModel(nn.Module):
     def __init__(self):
          super().__init__()
          self.x = 1

def forward(self, x):
     return 2*pow((pow(x,3)+1),(-1/2))

model = MyModel()
optimizer = optim.SGD(model.parameters(),lr=alpha)
terminationCond = False

while not terminationCond:
     f = model.forward(x)
     f.backward()
     optimizer.step()
     if x.grad < 0.001:
           terminationCond = True
     optimizer.zero_grad()

But I cannot output the correct value of x, how to modify my code in order to find a minizer of a function f?

Upvotes: 1

Views: 724

Answers (1)

Bhupen
Bhupen

Reputation: 1320

There are a few things that need to be considered ...

  1. For x to be a parameter (model.parameter()) x should be nn.Parameter(torch.as_tensor([1.]))
  2. You are passing what (and why) x to the forward method?
  3. The function f(x) = 2*(x^3+1)^(-1/2) is inversely proportional to x. When x goes up in value, the function goes down. The SGD will minimize the value of the function, hence maximize x.
f(0) = 2.0
f(1) = 1.41421
f(2) = 0.66666

Here is a working example of minimizing f(x) ie maximizing x.

import torch
import torch.nn as nn
from torch import optim

alpha = 0.1

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.x = nn.Parameter(torch.as_tensor([1.0]))
    def forward(self):
        return 2*pow((pow(self.x, 3)+1), (-1/2))

model = MyModel()
optimizer = optim.SGD(model.parameters(), lr=alpha)
terminationCond = False
print(model.x)

f = model()
f.backward()
optimizer.step()
optimizer.zero_grad()
print(model.x)

Upvotes: 1

Related Questions