CODEWITHSUNDEEP
pythonmachine-learningdeep-learningpytorch
Benjamin Crouzier
Benjamin Crouzier

Reputation: 41905

optim.lr_scheduler.ReduceLROnPlateau gives error value cannot be converted to type float without overflow: inf

I am using pytorch with this install command: pip3 install http://download.pytorch.org/whl/cu80/torch-0.3.1-cp35-cp35m-linux_x86_64.whl.

I have a model that trains without issues, but when I add a learning rate scheduler, I get an error

My scheduler:

# In init
self.optimizer = optim.Adam(self.model.parameters(), lr=0.01)
self.scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    self.optimizer, 'min', factor=0.1, patience=5, verbose=True
)

# after each epoch
self.scheduler.step(loss)

The error:

...
my_project/.env/lib/python3.5/site-packages/torch/optim/lr_scheduler.py in <lambda>(a, best)
    330         if mode == 'min' and threshold_mode == 'rel':
    331             rel_epsilon = 1. - threshold
--> 332             self.is_better = lambda a, best: a < best * rel_epsilon
    333             self.mode_worse = float('Inf')
    334         elif mode == 'min' and threshold_mode == 'abs':

RuntimeError: value cannot be converted to type float without overflow: inf

Doc: http://pytorch.org/docs/master/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau Related thread: https://discuss.pytorch.org/t/value-cannot-be-converted-to-type-double-without-overflow-inf/11752/7

Upvotes: 0

Views: 1821

Answers (1)

Benjamin Crouzier
Benjamin Crouzier

Reputation: 41905

I'm using gpu tensors, eg:

Variable(torch.from_numpy(X).type(torch.FloatTensor).cuda(), requires_grad=False)

If I cast it to the cpu like that, the error goes away

# after each epoch
self.scheduler.step(loss.cpu().data.numpy())

Still, I would like a cleaner solution.

Upvotes: 1

Related Questions