Using multi-output labels in keras ImageDataGenerator.flow() and using model.fit_generator()

Question

I have a single-input,multi-output Neural Network model whose last layers are

out1 = Dense(168, activation = 'softmax')(dense)
out2 = Dense(11, activation = 'softmax')(dense)
out3 = Dense(7, activation = 'softmax')(dense)

model = Model(inputs=inputs, outputs=[out1,out2,out3])

the Y-labels for each image are as follows

train
>>

              image_id    class_1   class_2  class_3    

0              Train_0         15         9        5    
1              Train_1        159         0        0
...
...
...
453651    Train_453651          0        15       34
453652    Train_453652         18         0        7

EDIT:-

train.iloc[:,1:4].nunique()
>>
class_1        168
class_2         11
class_3          7
dtype: int64

So looking at these different range of classes, should I use categorical_crossentropy or sparse_categorical_crossentropy? and how should I use the Y_labels in flow for the code given below?

imgs_arr = df.iloc[:,1:].values.reshape(df.shape[0],137,236,1)
# 32332 columns representing pixels of 137*236 and single channel images.
# converting it to (samples,w,h,c) format

Y = train.iloc[:,1:].values #need help from here

image_data_gen = ImageDataGenerator(validation_split=0.25)
train_gen = image_data_gen.flow(x=imgs_arr, y=Y, batch_size=32,subset='training')
valid_gen = image_data_gen.flow(x=imgs_arr,y=Y,subset='validation')

is this this the right way to pass Yor use Y=[y1,y2,y3] where

y1=train.iloc[:,1].values
y2=train.iloc[:,2].values
y3=train.iloc[:,3].values

Answer 1

Another workaround is:

1. Make an array of tuple, then pass it to the ImageDataGenerator flow method.
2. Make an iterator method that accepts the iterator made by the previous step. This iterator converts back the array of tuple to list of arrays.

Here is the methods to implement the steps above:

def make_array_of_tuple(tuple_of_arrays):
    array_0 = tuple_of_arrays[0]
    array_of_tuple = np.empty(array_0.shape[0], dtype=np.object)
    for i, tuple_of_array_elements in enumerate(zip(*tuple_of_arrays)):
        array_of_tuple[i] = tuple_of_array_elements
    return array_of_tuple

def convert_to_list_of_arrays(array_of_tuple):
    array_length = array_of_tuple.shape[0]
    tuple_length = len(array_of_tuple[0])
    array_list = [
        np.empty(array_length, dtype=np.uint8) for i in range(tuple_length) ]
    for i, array_element_tuple in enumerate(array_of_tuple):
        for array, tuple_element in zip(array_list, array_element_tuple):
            array[i] = tuple_element
    return array_list

def tuple_of_arrays_flow(original_flow):
    while True:
        (X, array_of_tuple) = next(original_flow)
        list_of_arrays = convert_to_list_of_arrays(array_of_tuple)
        yield X, list_of_arrays

To call the ImageDataGenerator flow() method and get the flow used for the model:

y_train = make_array_of_tuple((y_train_1, y_train_2, y_train_3))
orig_image_flow = train_image_generator.flow(X_train, y=y_train)
train_image_flow = tuple_of_arrays_flow(orig_image_flow)

The size of y_train is the same as X_train, so it should be accepted. 'train_image_flow' returns list of arrays that should be accepted by the Keras multi-output model.

ADDED (2019/01/26)

One another idea, simpler than the above one:

1. Pass array of indices, which contains 0, 1, 2, ..., to ImageDataGenerator.flow().
2. In the iterator, select the elements in the arrays for the multiple output by using the returned indices array from the original flow.

Here is the implementation:

def make_multi_output_flow(image_gen, X, y_list, batch_size):
    y_item_0 = y_list[0]
    y_indices = np.arange(y_item_0.shape[0])
    orig_flow = image_gen.flow(X, y=y_indices, batch_size=batch_size)

    while True:
        (X, y_next_i) = next(orig_flow)
        y_next = [ y_item[y_next_i] for y_item in y_list ]
        yield X, y_next

This is an example to call the method above.

y_train = [y_train_1, y_train_2, y_train_3]
multi_output_flow = make_multi_output_flow(
    image_data_generator, X_train, y_train, batch_size)

Answer 2

Ouch....

By the message given in your flow, you will need a single output. So you need to make the separation inside your model. (Keras failed to follow its own standards there)

This means something like:

Y = train.iloc[:,1:].values #shape = (50210, 3)

With a single output like:

out = Dense(168+11+7, activation='linear')(dense)

And a loss function that handles the separation:

def custom_loss(y_true, y_pred):
    true1 = y_true[:,0:1]
    true2 = y_true[:,1:2]
    true3 = y_true[:,2:3]

    out1 = y_pred[:,0:168]
    out2 = y_pred[:,168:168+11]
    out3 = y_pred[:,168+11:]

    out1 = K.softmax(out1, axis=-1)
    out2 = K.softmax(out2, axis=-1)
    out3 = K.softmax(out3, axis=-1)

    loss1 = K.sparse_categorical_crossentropy(true1, out1, from_logits=False, axis=-1)
    loss2 = K.sparse_categorical_crossentropy(true2, out2, from_logits=False, axis=-1)
    loss3 = K.sparse_categorical_crossentropy(true3, out3, from_logits=False, axis=-1)

    return loss1+loss2+loss3

Compile the model with loss=custom_loss.

Then the flow should stop complaining when you do flow.

Just make sure X and Y are exactly in the same order: imgs_arr[i] corresponds to Y[i] correctly.