CODEWITHSUNDEEP
pytorch
Alex I
Alex I

Reputation: 20287

Modify PyTorch model for inference - then resume training

I’d like to alternate inference and training in a model in PyTorch, but I need to modify it during inference, and I have some questions about that.

Can I unload the model from the gpu by calling model.to('cpu'), make a modified copy (and run it on the gpu), and then move the original back to gpu by calling model.to('gpu')? In other words, is moving the model gpu->cpu->gpu a lossless operation? What happens to the parameters that were passed to the optimizer? I don’t want to lose the optimizer state

What is the best way to make a copy of an in-memory model? I can save it and then reload a copy, but not sure if that is necessary just to copy.

If I want to run inference in half precision (more than 2x faster in this case), can I change the model to half and then change it back? Is that lossless? (Does the model keep a full-precision copy of everything, or does it replace weights with half-precision in place?)

The model is a ResNet50 lookalike. Not enough GPU memory for two models :)

Upvotes: 0

Views: 372

Answers (1)

dpetrini
dpetrini

Reputation: 1239

"What is the best way to make a copy of an in-memory model? I can save it and then reload a copy, but not sure if that is necessary just to copy."

You can copy a model with:

import copy
...
best_model = copy.deepcopy(model)

with best_model you can save to disk or load in other model, etc

Upvotes: 1

Related Questions