Lars Ericson
Reputation: 2094
Can I speed up inference in PyTorch using autocast (automatic mixed precision)?
The docs (see also this) for autocast in PyTorch only discuss training. Does it speed things up if I also use autocast for inference?
Upvotes: 3
Views: 3419
Answers (1)
Szymon Maszke
Reputation: 24815
Yes it could (may not in some cases though).
You are processing data with lower precision (e.g. float16 vs float32).
Your program has to read and process less data in this case.
This might help with cache locality and hardware specific software (e.g. tensor cores if using CUDA)
Upvotes: 3
Related Questions
- onnxruntime inference is way slower than pytorch on GPU
- is there any way to optimize pytorch inference in cpu?
- Pytorch Model Optimization: Automatic Mixed Precision vs Quantization?
- Mixed Precision(Pytorch Autocast) Slows Down the Code
- Speed up training deep learning model in pytorch
- PyTorch's nn.Conv2d with half-precision (fp16) is slower than fp32
- Pytorch mixed precision learning, torch.cuda.amp running slower than normal
- slow Inference time for Neural Net
- pytorch runs slow when data are pre-transported to GPU
- Dealing with large slowdown when moving PyTorch code to GPU