Layers error while trying to use keras_vggface

Question

I get this error with trying to run my code:

Traceback (most recent call last):
  File "/Users/kegak/AI_Project/main.py", line 24, in <module>
    vggface = VGGFace(model='resnet50', include_top=False, input_shape=(244, 244, 3))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/keras_vggface/vggface.py", line 94, in VGGFace
    return RESNET50(include_top=include_top, input_tensor=input_tensor,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/keras_vggface/models.py", line 286, in RESNET50
    model.load_weights(weights_path)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 2347, in load_weights
    hdf5_format.load_weights_from_hdf5_group(f, self.layers)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 688, in load_weights_from_hdf5_group
    raise ValueError('You are trying to load a weight file '
ValueError: You are trying to load a weight file containing 106 layers into a model with 53 layers.

Code here:

import matplotlib.image
import tensorflow as tf
import keras
from tensorflow.python.keras.layers import Flatten, Dense, Input
from keras_vggface.vggface import VGGFace
from tensorflow.python.keras.utils.data_utils import get_file
import keras_vggface.utils
import mtcnn
import matplotlib
import ssl

ssl._create_default_https_context = ssl._create_unverified_context


face_photo = matplotlib.image.imread('my_face.jpg')

my_face = mtcnn.MTCNN().detect_faces(face_photo)


dataset = tf.keras.preprocessing.image_dataset_from_directory('folder', shuffle=False, batch_size=8, image_size=(244,244))

data_aug = keras.Sequential([keras.layers.RandomFlip('horizontal')], keras.layers.RandomRotation(0.2))

vggface = VGGFace(model='resnet50', include_top=False, input_shape=(244, 244, 3))

vggface.trainable = False

final_layer = vggface.get_layer('avg_pool').output

inputs = tf.keras.Input(shape=(244, 244, 3))

x = data_aug(inputs)

x = vggface(x)

x = Flatten(name='flatten')(x)

output = Dense(2, name='classifier')(x)

custom_model1 = keras.Model(inputs, output)

custom_model1.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=['accuracy'])

custom_model1.fit(dataset, epochs=20)

x1, y1, width, height = my_face[0]['box']
x2, y2 = x1 + width, y1 + height
the_face = face_photo[y1:y2, x1:x2]

prob_model = keras.Sequential([custom_model1, tf.keras.layers.Softmax()])

prediction = prob_model(the_face)
print(prediction)

I have been following this tutorial, and have been runing into issues: https://www.width.ai/post/tensorflow-facial-recognition

I have edited the keras_vggface in accordance to this: https://github.com/rcmalli/keras-vggface/issues/97 And also had to change the files quite extensively due to the '/' character being in some strings.