CODEWITHSUNDEEP
python-3.xkerasconv-neural-network
Peter
Peter

Reputation: 117

Tuning neural network hyperparameters when using Keras functional API

I have a neural network that contains two branches. One branch takes input to a convolution neural network. And other branch is a fully connected layer. I merge these two branches and then get an output using softmax. I can not use a sequential model because it's deprecated and therefore, had to use functional API. I want to tune the hyperparameters for a convolutional neural network branch. For example, I want to figure out how many convolution layers I should use. If it was a sequential model I would've used a for loop but since I am using a functional API I can't really do that. I've attached my code. Could anyone tell me how can optimise my neural network for number of convolutions in a smart way instead of making a lot of different scripts with different number of convolution layers.

Suggestions would be appreciated.


i1 = Input(shape=(xtest.shape[1], xtest.shape[2]))

###Convolution branch
c1 = Conv1D(128*2, kernel_size=ksize,activation='relu',kernel_regularizer=keras.regularizers.l2(l2_lambda))(i1)
c1 = Conv1D(128*2, kernel_size=ksize, activation='relu',kernel_regularizer=keras.regularizers.l2(l2_lambda))(c1)
c1 = AveragePooling1D(pool_size=ksize)(c1)
c1 = Dropout(0.2)(c1)

c1 = Conv1D(128*2, kernel_size=ksize, activation='relu',kernel_regularizer=keras.regularizers.l2(l2_lambda))(c1)
c1 = AveragePooling1D(pool_size=ksize)(c1)
c1 = Dropout(0.2)(c1)

c1 = Flatten()(c1)

###fully connected branch
i2 = Input(shape=(5000, ))
c2 = Dense(64,  activation='relu',kernel_regularizer=keras.regularizers.l2(l2_lambda))(i2)
c2 = Dropout(0.1)(c2)


###concatenating the two branches
c = concatenate([c1, c2])

x = Dense(256, activation='relu', kernel_initializer='normal',kernel_regularizer=keras.regularizers.l2(l2_lambda))(c)
x = Dropout(0.25)(x)

###Output branch 
output = Dense(num_classes, activation='softmax')(x)

model = Model([i1, i2], [output])

model.summary()

With sequential models I can use a for loop so for example:


layers = [1,2,3,4,5]

b1 = Sequential()
b1.add(Conv1D(128*2, kernel_size=ksize,
                 activation='relu',
                 input_shape=( xtest.shape[1], xtest.shape[2]),
                 kernel_regularizer=keras.regularizers.l2(l2_lambda)))

for layer in layers:
    count = layer
    while count > 0:
        b1.add(Conv1D(128*2, kernel_size=ksize, activation='relu',kernel_regularizer=keras.regularizers.l2(l2_lambda)))
        count -= 1

b1.add(MaxPooling1D(pool_size=ksize))
b1.add(Dropout(0.2))

b1.add(Flatten())
b2 = Sequential()

b2.add(Dense(64, input_shape = (5000,), activation='relu',kernel_regularizer=keras.regularizers.l2(l2_lambda)))

for layer in layers:
    count = layer
    while count > 0:
    b2.add(Dense(64,, activation='relu',kernel_regularizer=keras.regularizers.l2(l2_lambda)))


model = Sequential()
model.add(Merge([b1, b2], mode = 'concat'))
model.add(Dense(256, activation='relu', kernel_initializer='normal',kernel_regularizer=keras.regularizers.l2(l2_lambda)))
model.add(Dropout(0.25))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
                  optimizer=keras.optimizers.Adam(),
                  metrics=['accuracy'])

Upvotes: 1

Views: 1159

Answers (2)

bmiselis
bmiselis

Reputation: 412

This is the minimal example of a model with a variable number of layers using Keras Functional API:

from keras.layers import Input, Conv2D, Dense, Dropout, Flatten, MaxPool2D
from keras.models import Model

def build_model(num_layers, input_shape, num_classes): 
  input = Input(shape=input_shape)
  x = Conv2D(32, (3, 3), activation='relu')(input)

  # Suppose you want to find out how many additional convolutional 
  # layers to add here.
  for _ in num_layers:
    x = Conv2D(32, (3, 3), activation='relu')(x)

  x = MaxPool2D((2, 2))(x)
  x = Flatten()(x)
  x = Dense(64, activation='relu')(x)
  x = Dropout(0.5)(x)
  x = Dense(num_classes, activation='softmax')(x)

  return Model(inputs=input, outputs=x)

model = build_model(num_layers=2, input_shape=(128, 128), num_classes=3)

These are the steps I would follow to find out how many 'middle' convolutional layers to use:

  1. Train several models with num_layers parameter set to various values. The code to build all those models is exactly the same, only the value of num_layers parameter changes across different training runs.
  2. Choose the one that has the best values of metrics you care about.

That's it!

Side note: as far as I know, Keras Sequential model isn't deprecated.

Upvotes: 1

josoler
josoler

Reputation: 1423

You can dynamically set your model structure using the functional API as well. For the convolutional branch you could use something like:

layer_shapes = (64, 64, 32)

for _ in layers:
    b1 = Conv1D(128*2, kernel_size=ksize, activation='relu', kernel_regularizer=keras.regularizers.l2(l2_lambda))(b1)

You just need to replace the Sequential.add by the corresponding variable assignment.

Upvotes: 1

Related Questions