keras generator with image and scalar

Question

I am trying to train some layers of a network whose inputs are an image and a scalar. Please see the figure below for a better understanding. .

As you can see only the dark yellow layers will be trained. So I need to freeze the rest, that is for later.

Purpose of this architecture is to map images (chest x-rays) to 14 kinds of diseases.

The images are stored in the following directory: /home/akde/chexnet/CheXNet-Keras/data/images

Names of the images are the image IDs.

A dataframe maps images (Images are named as the Image ID) to classes (diseases) As you can see an image can be mapped to more than one class (disease).

Another dataframe maps the images (Image IDs) to the patient age. You can see it below.

Image is the first input and patient age is the second.

So in short, for each image id, I have an image and age value which are in 2 separate dataframes.

I can already test (gives absurd results since the network is not trained, but still proves that the network accepts the input and gives some result) it using the following code.

res3 = model3.predict( [test_image, a] )

where a is the scalar input while the test_image is the image input.

My training data is stored in multiple dataframes, having read that post, I deduce that flow_from_dataframe should be used.

The first thing I have done was to see this post which explains how to use mixed inputs. That gave me some background but since it does not use fit_generator (instead uses fit) it did not solve my problem.

Then I have read this post which does not use multiple inputs. Again no clue.

Afterwards, I have seen this post, which takes 2 images as input ( not one image one scalar). So again no help.

Even though I haven't found a solution to my problem I have written the following piece of code which will be the skeleton the solution.

datagen=ImageDataGenerator(rescale=1./255., validation_split=0.25)

train_generator = datagen.flow_from_dataframe(traindf,
                                              directory="/home/akde/chexnet/CheXNet-Keras/data/images",
                                              class_mode="other",
                                              x_col="Image Index",
                                              y_col=["Atelectasis", "Cardiomegaly", "Effusion", "Infiltration", "Mass",
                                                     "Nodule", "Pneumonia", "Pneumothorax", "Consolidation", "Edema",
                                                     "Emphysema", "Fibrosis", "Pleural_Thickening", "Hernia"],
                                              color_mode="rgb",
                                              batch_size=32,
                                              target_size=(224, 224)
                                                     )

STEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size

model3.compile(optimizers.rmsprop(lr=0.0001, decay=1e-6),loss="categorical_crossentropy",metrics=["accuracy"])

model3.fit_generator(generator=train_generator,
                    steps_per_epoch=STEP_SIZE_TRAIN,
                    epochs=10
)

I know this piece of code is far from the solution.

So how can I create a generator that uses 2 dataframes which are explained earlier (the one that maps images to the diseases and the other one which maps image IDs to age).

In other words, what is the way of writing a generator that takes an image and a scalar value as an input, considering the fact that both are represented in dataframes. How can I write the generator that written in bold below.

model3.fit_generator(**generator=train_generator**,
                    steps_per_epoch=STEP_SIZE_TRAIN,
                    epochs=10
)

Answer 1

For your purpose you need to create a custom generator.

I will recommand you to take a deep look at this link :
https://blog.ml6.eu/training-and-serving-ml-models-with-tf-keras-3d29b41e066c

And especially this code :

import ast
import numpy as np
import math
import os
import random
from tensorflow.keras.preprocessing.image import img_to_array as img_to_array
from tensorflow.keras.preprocessing.image import load_img as load_img

def load_image(image_path, size):
    # data augmentation logic such as random rotations can be added here
    return img_to_array(load_img(image_path, target_size=(size, size))) / 255.

class KagglePlanetSequence(tf.keras.utils.Sequence):
    """
    Custom Sequence object to train a model on out-of-memory datasets. 
    """

    def __init__(self, df_path, data_path, im_size, batch_size, mode='train'):
        """
        df_path: path to a .csv file that contains columns with image names and labels
        data_path: path that contains the training images
        im_size: image size
        mode: when in training mode, data will be shuffled between epochs
        """
        self.df = pd.read_csv(df_path)
        self.im_size = im_size
        self.batch_size = batch_size
        self.mode = mode

        # Take labels and a list of image locations in memory
        self.wlabels = self.df['weather_labels'].apply(lambda x: ast.literal_eval(x)).tolist()
        self.glabels = self.df['ground_labels'].apply(lambda x: ast.literal_eval(x)).tolist()
        self.image_list = self.df['image_name'].apply(lambda x: os.path.join(data_path, x + '.jpg')).tolist()

    def __len__(self):
        return int(math.ceil(len(self.df) / float(self.batch_size)))

    def on_epoch_end(self):
        # Shuffles indexes after each epoch
        self.indexes = range(len(self.image_list))
        if self.mode == 'train':
            self.indexes = random.sample(self.indexes, k=len(self.indexes))

    def get_batch_labels(self, idx): 
        # Fetch a batch of labels
        return [self.wlabels[idx * self.batch_size: (idx + 1) * self.batch_size],
                self.glabels[idx * self.batch_size: (idx + 1) * self.batch_size]]

    def get_batch_features(self, idx):
        # Fetch a batch of images
        batch_images = self.image_list[idx * self.batch_size: (1 + idx) * self.batch_size]
        return np.array([load_image(im, self.im_size) for im in batch_images])

    def __getitem__(self, idx):
        batch_x = self.get_batch_features(idx)
        batch_y = self.get_batch_labels(idx)
return batch_x, batch_y

Hope this will help to find your solution !