The activation in my CNN does not look correct - or is the heatmap the problem?

Question

I am producing heatmaps for my convolutional neural networks made via Keras, as described here. When I run that algorithm for a vanilla VGG16 net, the heatmap looks fine:

Then I created my own custom model, based on that VGG16 network, but with custom top layers:

input_layer = layers.Input(shape=(img_size, img_size, 3), name="model_input")
vgg16_base = VGG16(weights="imagenet", include_top=False, input_tensor=input_layer)
temp_model = vgg16_base.output
temp_model = layers.Flatten()(temp_model)
temp_model = layers.Dense(256, activation="relu")(temp_model)
temp_model = layers.Dense(1, activation="sigmoid")(temp_model)
custom = models.Model(inputs=input_layer, outputs=temp_model)

However, when I generate the heatmap for the same layer of my own custom network (i.e. the last conv. layer from the VGG16 base, being part of my new network), using the very same code/function, the heatmap does not look right:

The validation/testing accuracy of my custom network is at 97-98%, so I would assume it works fine. How come that the activation/heatmap is so off then? Or do I miss something else here?

PS: For your reference, the heatmap is created via the function listed here. It is called like this:

# Load the image from disk and preprocess it via Keras tools
img_path = "/path/to/image.jpg"
img = image.load_img(img_path, target_size=(224, 224))
img_tensor = image.img_to_array(img)
img_tensor = np.expand_dims(img_tensor, axis=0)
img_tensor = preprocess_input(img_tensor)

# At this point I either load the VGG16 model directly (heatmapü works),
# or I create my own custom VGG16-based model (heatmap does not work)
# The model itself is then stored into the variable "model"

preds = model.predict(img_tensor)
model_prediction = model.output[:, np.argmax(preds[0])]

# Then I call the custom function referred to above
input_layer = model.get_layer("model_input")
conv_layer = model.get_layer("block5_conv3")
plot_conv_heat_map(model_prediction, input_layer, conv_layer, img_tensor, img_path)

Answer 1

Short answer: Since you have labeled a dog as 1 and a cat as 0 in training phase, you need to replace model_prediction with 1 - model_prediction to find the areas relevant to a cat:

plot_conv_heat_map(1 - model_prediction, ...)

---

Long answer: When you are using the original VGG model there are 1000 neurons (assuming you are using the pre-trained ImageNet model) in the last layer, one for each of the 1000 distinct classes:

# last layer in VGG model
x = layers.Dense(classes, activation='softmax', name='predictions')(x)

Each of those neurons have an output value from zero to one (with the condition that the sum of outputs must be one). So the most activated neuron (i.e. the one with the highest output) corresponds to the predicted class. So you find it like this:

model_prediction = model.output[:, np.argmax(preds[0])]
                                        \
                                         \___ finds the index of the neuron with maximum output

and then you pass it to the visualization function to compute its gradient with respect to the selected convolution layer and visualize the heat map:

plot_conv_heat_map(model_prediction, ...)

So far, so good. However, in your custom model you have transformed the problem from a multi-class classification task into a binary classification task, i.e. dog vs cat. You are using a sigmoid layer with one single unit as the last layer and have considered the active state (i.e. outputs of near 1) of the neuron as dog and the inactive state (i.e. outputs of near 0) of the neuron as cat. So your network is essentially a dog detector, and if there is no dog then we assume there is a cat in the image.

All right, you might ask "what's the problem with that?" The answer is there is no problem in terms of training of the model and, as you have suggested, you got a good training accuracy. However, remember the assumption behind the visualization function: it takes as input the neuron with the highest output which corresponds to the class detected in the image. So, given a cat image to your custom model the output of last layer would be a very low number, say 0.01. So one interpretation of that number is that the probability that this image is a dog is 0.01. So what happens when you give it directly to your visualization function? Yeah, you guessed it: it would find all the areas in the image which are most relevant to a dog! You might still object "But I have given it a cat image!!!" It does not matter, since that neuron activates when a dog is present and therefore when you take its gradient with respect to the convolution layer, the areas most relevant to a dog would be highly represented and shown in the heatmap. However, the visualization would be correct if you give the model a dog image.

"So what should we do when when want to visualize the areas most relevant to a cat?" It's easy: just make that neuron a cat detector. "How?" Just create its complement: 1 - model_prediction. This gives you the probability that a cat is present in the image. And you can use it easily like this to plot the cat-relevant areas in the image:

plot_conv_heat_map(1 - model_prediction, ...)

Alternatively, you can change the last layer of your model to have 2 neurons with a softmax activation and then re-train it:

temp_model = layers.Dense(2, activation="softmax")(temp_model)

This way each of the classes, i.e. dog and cat, would have its own neuron and therefore no problem would occur when visualizing the activation heat-map.