CODEWITHSUNDEEP
ioscoremlcoremltools
Artem Shalagin
Artem Shalagin

Reputation: 157

Running CoreML batchnorm and padding on ANE

I'm trying to optimize my PyTorch model to use it on iOS. I'm using coremltools to convert the model to CoreML and XCode 14 beta/iOS 16 beta in order to use new Performance Instruments from Apple.

I'm currently having troubles with two types of layers: BatchNorm and general_padding.

The Performance report shows that these layers are not available on the Apple Neural Engine (ANE) so the model mostly runs on GPU which makes it slow.

I downloaded several models from Apple model library (FCRN model for example) and profiled them. While the General tab shows that these models have a lot of BatchNorm layers, the Performance tab doesn't show any BatchNorm layers at all and runs the whole model on ANE.

Why can this happen, are there different types of batchnorm layers or something?

I noticed that most batchnorm layers are not supported on the ANE but several are marked as supported on ANE even though they seem to be similar.

Also it looks like padding is not supported on ANE at all, is that true that I should remove all padding from the model architecture in order for it to run on ANE?

Upvotes: 1

Views: 266

Answers (1)

Artem Shalagin
Artem Shalagin

Reputation: 157

Turns out XCode's Profiling tool shows InstanceNorm layers as BatchNorm layers. InstanceNorm cannot run on ANE, hence the slowdown. Also BatchNorm can be fused into the previous convolution layer in many cases, that's why most BatchNorm layers don't show up in the Performance report.

Upvotes: 0

Related Questions