CODEWITHSUNDEEP
machine-learningdeep-learningconv-neural-networkflops
Manav Madan
Manav Madan

Reputation: 71

Calculating the Number of flops for a given Neural Network?

I have a neural network (ALEXnet or VGG16) written with Keras for Image Classification and I would like to calculate the number of floating point operations for a network. The size of the images in the dataset could vary. Can a generalized code be written in python which could calculate flops automatically? Is there any library available?

I am working with spyderAnaconda and the defined network is a sequential model.

Upvotes: 6

Views: 14252

Answers (3)

cinkyoung
cinkyoung

Reputation: 21

many papers using their own flops counting code. it is made by entering the input size of certain operation's tensor. so they manually calculate it's flops

you can find it with keyword like 'flops constraint' or 'flops counter' in github.

or there are 'torchstat' tool which counts the flops and memory usage etc.

Upvotes: 1

Lawrence Francis
Lawrence Francis

Reputation: 61

FLOPs are the floating-point operations performed by a model. It is usually calculated using the number of multiply-add operations that a model performs. Multiply-add operations, as the name suggests, are operations involving multiplication and addition of 2 or more variables. For example, the expression, a * b + c * d, has 2 flops while a * b + c * d + e * f + f * h has 4 flops.

Let's now take a simple linear regression model for example. Assume this model has 4 parameters w1, w2, w3, and w4 and a bias b0. Inference on an input data, X = [x1, x2, x3, x4] results in output = x1 * h1 + x2 * h2 + x3 * h3 + x4 * h4 + b0. This operation has 4 flops

The FLOPs measurement in CNNs involves knowing the size of the input tensor, filters and output tensor for each layer. Using this information, flops are calculated for each layer and added together to obtain the total flops. Let's look at the first layer in VGG16 with input tensor of size 224x224x3, 64 filters of size 3x3x3 and output size of 224x224x64. Each element in the output results from an operation involving (3x3x3) multiply-add between the filter and the input tensor. Hence the number of flops for the first layer of VGG16 is (3x3x3)x(224x224x64)= 86,704,128

Upvotes: 6

Prune
Prune

Reputation: 77850

There is no such code because the quantity of FLOPs is dependent on the hardware and software implementations. You can certainly derive a typical quantity from expanding the layer-by-layer operations for each parameter and weight. and making reasonable implementation assumptions for each activation function.

Input dimensions will proportionately affect the computations for the first layer.

I'm not sure what you intend for "generalized code in Python"; do you envision using a form of the Keras model as the input? This is possible, but you need to write modules that will extract the kernel characteristics and connection logic from the Keras representation.

Your quantity of operations will vary from one implementation to another. Hardware architectures now directly support parallel operations and short-cuts for sparse matrices. Some have extra functionality for adjusting floating-point representations for greater training speed. Software platforms include control and data flow analysis to optimize the functional flow. Any of these will change the FLOPs computation.

Upvotes: -1

Related Questions