TimeDistributed layer to apply several convolutional layers error

Question

I have an issue with the tf.keras.layers.TimeDistributed layer (https://www.tensorflow.org/api_docs/python/tf/keras/layers/TimeDistributed).

I am aware that TimeDistributed can be used to apply a single layer (dense, convolutional...) to a set of inputs, obtaining a set of outputs.

Not only that, but until recently I was able to use it to apply an entire "submodel" to all the inputs. That is, a series of layers, not just one. An example of this is explained here by Patrice Ferlet (https://medium.com/smileinnovation/training-neural-network-with-image-sequence-an-example-with-video-as-input-c3407f7a0b0f).

Using that source as example I can define a sequential "submodel" like this:

import keras
from keras.layers import Conv2D, BatchNormalization, \
MaxPool2D, GlobalMaxPool2D

def build_convnet(shape=(112, 112, 3)):
  momentum = .9
  model = keras.Sequential()
  model.add(Conv2D(64, (3,3), input_shape=shape,
      padding='same', activation='relu'))
  model.add(Conv2D(64, (3,3), padding='same', activation='relu'))
  model.add(BatchNormalization(momentum=momentum))

  model.add(MaxPool2D())

  model.add(Conv2D(128, (3,3), padding='same', activation='relu'))
  model.add(Conv2D(128, (3,3), padding='same', activation='relu'))
  model.add(BatchNormalization(momentum=momentum))

  model.add(MaxPool2D())

  model.add(Conv2D(256, (3,3), padding='same', activation='relu'))
  model.add(Conv2D(256, (3,3), padding='same', activation='relu'))
  model.add(BatchNormalization(momentum=momentum))

  model.add(MaxPool2D())

  model.add(Conv2D(512, (3,3), padding='same', activation='relu'))
  model.add(Conv2D(512, (3,3), padding='same', activation='relu'))
  model.add(BatchNormalization(momentum=momentum))

  # flatten...
  model.add(GlobalMaxPool2D())
  return model

And then include this submodel inside a superior model which calls TimeDistributed with the whole initial submodel (convnet).

from keras.layers import TimeDistributed, GRU, Dense, Dropout

def action_model(shape=(5, 112, 112, 3), nbout=3):
  # Create our convnet with (112, 112, 3) input shape
  convnet = build_convnet(shape[1:])

  # then create our final model
  model = keras.Sequential()
  # add the convnet with (5, 112, 112, 3) shape
  model.add(TimeDistributed(convnet, input_shape=shape))
  # here, you can also use GRU or LSTM
  model.add(GRU(64))
  # and finally, we make a decision network
  model.add(Dense(1024, activation='relu'))
  model.add(Dropout(.5))
  model.add(Dense(512, activation='relu'))
  model.add(Dropout(.5))
  model.add(Dense(128, activation='relu'))
  model.add(Dropout(.5))
  model.add(Dense(64, activation='relu'))
  model.add(Dense(nbout, activation='softmax'))
  return model

Now this works well, and I can get the model structure calling

mod=action_model()
mod.summary()

But if instead of this I define the convnet model using as backbone a predefined arquitecture from keras, like VGG16, there seems to be an error. (I also needed to change as well keras.Sequential by tf.keras.models.Sequential)

import tensorflow as tf
from keras.layers import Flatten

def build_convnet():

    prevModel = tf.keras.applications.vgg16.VGG16(
        include_top=False,
        input_shape=(112, 112, 3),
        weights='imagenet'  # ImageNet weights
    )

    model = tf.keras.models.Sequential()

    model.add(prevModel)
    model.add(Flatten())

    return model

def action_model(shape=(5, 112, 112, 3), nbout=3):
  # Create our convnet with (112, 112, 3) input shape
  convnet = build_convnet()

  # then create our final model
  model = tf.keras.models.Sequential()
  # add the convnet with (5, 112, 112, 3) shape
  model.add(TimeDistributed(convnet, input_shape=shape))
  # here, you can also use GRU or LSTM
  model.add(GRU(64))
  # and finally, we make a decision network
  model.add(Dense(1024, activation='relu'))
  model.add(Dropout(.5))
  model.add(Dense(512, activation='relu'))
  model.add(Dropout(.5))
  model.add(Dense(128, activation='relu'))
  model.add(Dropout(.5))
  model.add(Dense(64, activation='relu'))
  model.add(Dense(nbout, activation='softmax'))
  return model

When I run this after defining my VGG16-based architecture

mod=action_model()
mod.summary()

I get the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
 in ()
----> 1 mod=action_model()
      2 mod.summary()

1 frames
/usr/local/lib/python3.7/dist-packages/keras/layers/wrappers.py in __init__(self, layer, **kwargs)
    121           'Please initialize `TimeDistributed` layer with a '
    122           '`tf.keras.layers.Layer` instance. You passed: {input}'.format(
--> 123               input=layer))
    124     super(TimeDistributed, self).__init__(layer, **kwargs)
    125     self.supports_masking = True

ValueError: Please initialize `TimeDistributed` layer with a `tf.keras.layers.Layer` instance. You passed:

So now it appears like python is complaining that I am using an input for TimeDistributed that is not a single layer. This doesn't make any sense, since the initial example works well and also involves using several layers with TimeDistributed. Apart from that, the VGG16 model also worked fine some weeks ago.

I am running all of this in Google CoLab.

Could someone help me figure out what is going on here? Is this caused by the new tensorflow 2.5.0 version? Everywhere I look I see people using TimeDistributed to apply a single layer, but applying a whole sequential model worked just fine until now (despite no apparent mention in the documentation).

Thank you!

TimeDistributed layer to apply several convolutional layers error

Answers (1)

Related Questions