CODEWITHSUNDEEP
pythontensorflowmachine-learningkeraskeras-layer
santasmic
santasmic

Reputation: 353

Loading a saved model in Keras with a custom layer and prediction results are different?

My network was achieving 96% accuracy on my dataset (edit: on predicting 9 classes). I saved the entire model for every epoch (weights included) whenever I ran it. I ran it 3 times, each time testing different hyperparameters, each time achieving around 96% accuracy.

When I try to load any of these tests now and run it again, it achieves around 50% accuracy. I'm extremely confident I'm running it on the same dataset.

Here's the interesting thing: if I train a new network instance of the exact same architecture, same size, shape, etc., it only ever reaches a max of 85% accuracy. Additionally, saving and loading these new training models works correctly, as in the model will reach the same 85% accuracy.

So, there's no problem with loading and no problem with my dataset. The only way this could happen is if something is wrong in my custom layer, or something else is happening.

Unfortunately I haven't been committing all of my changes to the custom layer to git. While I can't guarantee that my custom layer is the exact same, I'm almost completely confident it is.

Any ideas to what could be causing this discrepancy?

Edit: To add more context, the layer is ripped from the ConvLSTM2d class but I replaced the call() function to simply be a vanilla RNN that uses convolution instead of the dot product. I am confident that the call() function is the same as before, but I am not confident that the rest of the class is the same. Is there anything else in the class that could affect performance? I've checked activation function already.

Upvotes: 2

Views: 1353

Answers (2)

santasmic
santasmic

Reputation: 353

I finally found the answer.

I have implemented a custom Lambda layer to handle reshaping. This layer has difficulties loading. Specifically, it reshapes one dimension into two dimensions on an arbitrary interval. The interval was defaulting to one specific value every time I loaded the model, even though it was incorrect. When I force it to be the correct interval, it works.

Upvotes: 1

ixeption
ixeption

Reputation: 2060

Well, let´s think about it, there are basically two possible reasons for your problem. Your custom layer is not the same and therefore some weights are not set up correctly, which leads to missclassification. In case of a binary classification you will end up in 50% accuracy on balanced data.

The second possible reason is, that threre is a bug in keras serialization. In this case your new trained and saved models should also show this issue, after deserialization.

While I can't guarantee that my custom layer is the exact same, I'm almost completely confident it is.

So the problem is not reproducable with a new model, you accuracy is 50% and you cannot guarantee that the custom layer matches. -> I guess it is you custom layer.

Upvotes: 1

Related Questions