How can I make an end-to-end 3D-2D CNN in Keras?

Question

I have two CAE models, one in 3D and the other in 2D. This 2D CAE takes the new representation generated by the first as input. My goal is to figure out how to combine them so that I can have an end-to-end full 3D-2D CAE model and how can I train it ?

Here is the code of each model :

#3D CAE (I have just implemented the first encoding part since my aim is to generate the new representation z)

in_3D = Input((100,100, 288, 1))
model_3D = Conv3D(8, (5, 5, 5), activation='relu', padding='same')(in_3D)
model_3D = MaxPooling3D((2, 2, 2), strides=(1, 1, 4), padding='same')(model_3D)
model_3D = Reshape((10000,72*8))(model_3D)
model_3D = Dense(350, activation="relu")(model_3D)
model_3D = Dense(250, activation="relu")(model_3D)
model_3D = Dense(198, activation="relu")(model_3D)
model_3D = Reshape((100,100, 198))(model_3D)
z = Permute((3,2, 1))(model_3D)

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 100, 100, 288, 1  0         
                             )]                                  
                                                                 
 conv3d_1 (Conv3D)           (None, 100, 100, 288, 8)  1008      
                                                                 
 max_pooling3d_1 (MaxPooling  (None, 100, 100, 72, 8)  0         
 3D)                                                             
                                                                 
 reshape (Reshape)         (None, 10000, 576)        0         
                                                                 
 dense (Dense)            (None, 10000, 350)        201950    
                                                                 
 dense_1 (Dense)            (None, 10000, 250)        87750     
                                                                 
 dense_2 (Dense)            (None, 10000, 198)        49698     
                                                                 
 reshape_1 (Reshape)         (None, 100, 100, 198)     0         
                                                                 
 permute (Permute)           (None, 198, 100, 100)     0         
                                                              

And the second 2D CAE model that recieves as input the new z (198,100,100) generated by the first model. Here 198 is passed as None

#2D CAE  

in_2D = Input((100,100, 1))
model_2D= Conv2D(16, (3, 3), activation='relu', padding='same', name='Conv1')(in_2D)
model_2D = MaxPooling2D((2, 2), padding='same')(model_2D)
model_2D = Flatten()(model_2D)
model_2D = Dense(48, activation='relu')(model_2D)
model_2D = Dense(36, activation='relu')(model_2D)
model_2D = Dense(12)(model_2D)
model_2D= Dense(100*100, activation='linear')(model_2D)

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 100, 100, 1)]     0         
                                                                 
 Conv1 (Conv2D)              (None, 100, 100, 16)      160       
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 50, 50, 16)       0         
 2D)                                                             
                                                                 
 flatten (Flatten)          (None, 40000)             0         
                                                                 
 dense (Dense)              (None, 48)                1920048   
                                                                 
 dense_1 (Dense)             (None, 36)                1764      
                                                                 
 dense_2 (Dense)             (None, 12)                444       
                                                                 
 dense_3 (Dense)            (None, 10000)             130000    

Any help will be much appreciated.

Answer 1

To combine both models, you can start by declaring them individually like so:

import keras
import tensorflow as tf
from tensorflow.keras.layers import *

in_3D = Input((100,100, 288, 1))
model_3D = Conv3D(8, (5, 5, 5), activation='relu', padding='same')(in_3D)
model_3D = MaxPooling3D((2, 2, 2), strides=(1, 1, 4), padding='same')(model_3D)
model_3D = Reshape((10000,72*8))(model_3D)
model_3D = Dense(350, activation="relu")(model_3D)
model_3D = Dense(250, activation="relu")(model_3D)
model_3D = Dense(198, activation="relu")(model_3D)
model_3D = Reshape((100,100, 198))(model_3D)
z = Permute((3,2, 1))(model_3D)
cae_model_3D = keras.Model(in_3D, z)

in_2D = Input((100,100, 1))
model_2D= Conv2D(16, (3, 3), activation='relu', padding='same', name='Conv1')(in_2D)
model_2D = MaxPooling2D((2, 2), padding='same')(model_2D)
model_2D = Flatten()(model_2D)
model_2D = Dense(48, activation='relu')(model_2D)
model_2D = Dense(36, activation='relu')(model_2D)
model_2D = Dense(12)(model_2D)
model_2D= Dense(100*100, activation='linear')(model_2D)
cae_model_2D = keras.Model(in_2D, model_2D)

Then declaring a combined model that passes the output of the first model as input to the second:

combined_model_input = Input((100, 100, 288, 1))
cae_model_3D_output = cae_model_3D(combined_model_input)
cae_model_3D_output_reshaped = tf.reshape(cae_model_3D_output, (-1, 100, 100, 1))
combined_model_output = cae_model_2D(cae_model_3D_output_reshaped)
combined_model = keras.Model(combined_model_input, combined_model_output)

Note that we had to reshape the output of the first model for it to consistent with the idea that you wanted 198 to be passed as the batch dimension (None).

The simplest way to train the model would be to call it's compile and fit methods. The exact parameters you pass to these methods will depend on the problem you are trying to solve and your own preferences. Here is a link to the official docs for help.