CODEWITHSUNDEEP
pythontensorflowkerasneural-network
machine_apprentice
machine_apprentice

Reputation: 439

How can I pretrain for a multi-input DNN for Keras

I'm currently developing a model using Keras + Tensorflow in order to determine the temperature range of a set of proteins. What I first did was create a pre-trained model that converts the proteins into embeddings and then predicts its respective temperature.

What I want to do now is incorporate this pre=trained model to a new model which can use this given model and respective weights as input. Then fit on a new dataset and predict once again. The following code for the new top model is:

UPDATED CODE

'Load Pretrained Model'
loaded_model = keras.models.load_model('pretrained_model')

#Freeze all model layer weights
loaded_model.trainable = False

input1 = np.expand_dims(x_train['input1'],1)
input2 = np.expand_dims(x_train['input2'], 1)
input3 = x_train['input3']

#Redefine Input Layers for ANN
input1 = Input(shape = (input1.shape[1],), name = "input1")
input2 = Input(shape = (input2.shape[1],), name = "input2")
input3 = Input(shape = (input3.shape[1],), name = "input2")

base_inputs = [input1, input2, input3]

x = loaded_model(base_inputs, training = False)
x = Dense(64, activation = "relu",  kernel_regularizer=regularizers.l2(0.01))(x)

output = Dense(1, activation = "sigmoid")(x)

top_model = Model(inputs = base_inputs, outputs = output)

# Compile the Model
top_model.compile(loss='mse', optimizer = Adam(lr = 0.0001), metrics = ['mse'])

This is not working correctly and I'm not sure on how to get this up and running. I'm struggling a bit to get this and come across this error quite often:

AttributeError: 'Dense' object has no attribute 'shape'

Any thoughts?

Upvotes: 1

Views: 319

Answers (2)

Ipvikukiepki-KQS
Ipvikukiepki-KQS

Reputation: 137

Could you please try to use the initialize the inputs using the keras layers and try? You have initialized the input shapes using Numpy. But, If I am right unfortunately the dense layer which you have imported from keras does not support this ('Dense' object has no attribute 'op'). Kindly note that, 'Input' is a keras layer

Could you try to as specified in the following link to initialize the keras inputs (https://keras.io/guides/functional_api/)?

As an Example,

input1 = keras.Input(shape=(1,))
input2 = keras.Input(shape=(1,))
input3 = keras.Input(shape=(1,))

It totally depends on your machine learning architecture whether to make layers trainable or not. In Case of transfer learning, You can just use the trained weights from a pre-trained model and train your new network using the trained weights acquired from that model. In this case, you have to freeze the layers of the pre-trained model. Hence trainable = False. You use these weights in the mathematical calculation of the hidden layers you will use in your custom architecture.

But from your code snippet, I could predict that you are not using any hidden layers like LSTM, RNN or any other cells for your sequential data. Also, you are trying to provide the initialized Numpy inputs to a pre-trained model. I don't think whether it is a right way to do so. From the code snippet, you are making the layer not trainable, but you are then trying to train the model.

Also, If I am right, I think that you have to train with new set of data using the Pre-trained model right? If so, then kindly look at the following link(https://keras.io/guides/transfer_learning/).

Upvotes: 2

Ipvikukiepki-KQS
Ipvikukiepki-KQS

Reputation: 137

Considering your problem, I could suggest that transfer learning approach would be a feasible solution. In transfer learning, you could use the trained model from one domain of a set of data to train similar kind of problem using other set of data. To clearly understand the how to make the layers trainable and freeze it, and fine-tuning can be understood in the following link(https://keras.io/guides/transfer_learning/)

Concerning the Attribute error, it is recommended to have a look at the following link (https://keras.io/guides/functional_api/)

At first, you to initialize the input node for keras along with the shape of the inputs with respect to the data you will feed to the train the model. An Example is shown below as follows,

inputs = keras.Input(shape=(784,))

or it can be something like as follows, if you are providing the image data

img_inputs = keras.Input(shape=(32, 32, 3))

The dense layer expects the input should be in a specific shape, which you can find according to your data. If you are not sure about it, please analyse the data at first. It will give you lot of information to proceed further.

Upvotes: 1

Related Questions