Combining two Pre Trained models with 2 different classes (dataset) for Prediction

Question

I have two Pre-Trained and saved Inception Models.

Model 1 = Inception Model with Imagenet Classes - It can predict an image in 1000 classes.

Model 2 = Inception Model with my own classiciation (20 Classes) - It can predict an image in 20 classes. Performed transfer learning and saved the model.

I would like to combine this both to predict images with 1020 Classes.

Model1 = inception_v3.InceptionV3(weights='imagenet')

Predicts image in 1000 classes

predictions1 = Model1.predict(processed_image)

Model2 = InceptionV3(weights='imagenet',
              include_top=False,
              input_shape=(224, 224, 3))

I have performed transfer learning with my own 20 classes. Same input shape for both models. Predicts image in 20 classes

predictions = Model2.predict_classes(precessed_image)

How do I combine 2 Pre-Trained Inception Model to predict Imagenet classes (1000) and my own classifiers (20 Classes) = predict images on 1020 classes?

Please give me your solution with a small snippet(code) as an example for better understanding. I am pretty new to Keras.

Answer 1

you can combine every two or more pretrained models, the below is my code for combining two Xception and inceptionResnet models which uses them for classification:

`"""# **InceptionResnet Model**"""
incp_resnet_base_model=InceptionResNetV2(weights="imagenet",include_top=False,input_shape =(224, 224,3))
incp_resnet_base_model.summary ()

for layer in incp_resnet_base_model.layers:
    layer.trainable = False

#get weights and features from base model    
incp_resnet_features = incp_resnet_base_model.output


"""# **Xception Model**"""

xception_base_model=Xception(weights="imagenet",include_top=False,input_shape =(224, 224,3))
xception_base_model.summary ()

for layer in xception_base_model.layers:
    layer.trainable = False

#get weights and features from base model    
xception_features = xception_base_model.output

"""# **Flatten features**"""

xception_features=Flatten(name="flatten_xception")(xception_features)
incp_resnet_features=Flatten(name="flatten_inceptionResnet")(incp_resnet_features)

"""# ** concat models outpur **"""
all_features = concatenate([xception_features, incp_resnet_features])

hybridModel = Dense(500, activation="relu")(all_features)
hybridModel = Dropout(.2)(hybridModel)
hybridModel = Dense(200, activation="relu")(hybridModel)
hybridModel = Dropout(.2)(hybridModel)
hybridModel = Dense(100, activation="relu")(hybridModel)
hybridModel = Dense(1, activation="sigmoid")(hybridModel)
final_model = Model(inputs=[xception_base_model.input,incp_resnet_base_model.input], outputs=hybridModel,name="final output")
final_model.compile(optimizer='Adam', loss=tensorflow.keras.losses.binary_crossentropy,metrics="accuracy" )
model_hist=final_model.fit([X_train,X_train], y_train,
                           validation_data=([X_valid,X_valid], y_valid), epochs=20,batch_size=200)

Answer 2

You would like to combine the pretrained model labels with your own labels, or in other words you are augmenting the pretrained model with new classes. The practical approach is to use the basis of transfer learning.

But let me tell you this, this is still a hot research topic. It is easier to retrain with your own classes than to add additional classes. Hard, not impossible!

What you should be doing: One way to do this is to change the last softmax layer to identify more classes than it is designed to label. Network surgery. You will have to train the model again and it will take more time.

Another way is to create a new custom model with all 1020 labels and train it on the whole dataset, which is not very efficient and you cannot exploit the weights from the pretrained model and you have to perform full training again.

A hack can be to use the checkpoint which already predicts 1000 classes. Add your data for the new class. Now you should combine the new class' data along with Imagenet data set, create TFRecords for all 1020 classes and train from the network checkpoint.

What you are trying to do is called "learning without forgetting". Please refer to the below paper for more information about how to implement this.

https://arxiv.org/abs/1606.09282

And the matlab code is available here.

https://github.com/lizhitwo/LearningWithoutForgetting

You may also try tweaking the below file to get the result want.

https://github.com/tensorflow/hub/blob/master/examples/image_retraining/retrain.py

Additionally, To get more information on retraining with pretrained models refer to the below link.

https://www.tensorflow.org/tutorials/images/hub_with_keras

Now coming to the answer,

How do we do it:

Rebuild the same model with increased number of labels in the final classification layer then restore all weights from the pretrained inception V3 except the final layer and fine tune the model.

We need specify two flags for this:

1. --pretrained_model_checkpoint_path
2. --fine_tune

The code will looks like this.

# Build the model. Note that we need to make sure the TensorFlow is ready to
# use before this as this command will not build TensorFlow.
cd tensorflow-models/inception
bazel build //inception:custom_train

# Path to the downloaded Inception-v3 model.
MODEL_PATH="${INCEPTION_MODEL_DIR}/inception-v3/model.ckpt-1456935"

# Directory where the data resides.
DATA_DIR=/tmp/custom-data/

# Directory where to save the checkpoint and events files.
TRAIN_DIR=/tmp/custom_train/

# Run the fine-tuning on the flowers data set starting from the pre-trained
# Imagenet-v3 model.
bazel-bin/inception/flowers_train \
  --train_dir="${TRAIN_DIR}" \
  --data_dir="${DATA_DIR}" \
  --pretrained_model_checkpoint_path="${MODEL_PATH}" \
  --fine_tune=True \
  --initial_learning_rate=0.001 \
  --input_queue_memory_factor=1

Refer to the below documentation for more information.

https://github.com/tensorflow/models/tree/master/research/inception

Answer 3

There is no right answer here. What you can do is for example, run the image through both classifiers and see whose best prediction has higher confidence. Other option could be creating a simple classifier through which you will run your image and whose prediction will be binary and will tell you which model to use (1-inception, 0-your model). Another option is to see the prediction distribution. For example, if you run some image through your classifier, if the object in an image is not in one of those 20 classes, prediction distribution will be scattered approximately equally between few classes with none of them standing out, all of them will have more or less the same confidence. That usually means that the image is not in the scope of the classifier - the model doesn't know where to put an image because that's not an object it has seen before so it cannot decide.

EDIT

Let's say model is the simple binary classifier that classifies images for inception in class 1 and images for your model in class 0. Then you could do something like this:

# open image
img = Image.open(path_to_image)

if model.predict(img): # positive case (classifier classified img to class '1')
  prediction = inception_model.predict(img)
else: # negative case (classifier classified img to class '0')
  prediction = my_model.predict(img)

Now the result of the prediction is in variable prediction. Here I used that simple classifier (model) to predict which model will actually be used to classify the image (inception or your model). I inception has been used, prediction will be a 1000-dimensional vector and if my_model has been used, prediction will be a 20-dimensional vector. But it doesn't matter because you can just get the highest value of prediction regardless of the vector size and work with that.