Keras fit generator - ValueError: Failed to find data adapter that can handle input

Question

i'm trying to fit my deep learning model with a custom generator.

When i fit the model, it shows me this error:

I tried to find similar questions, but all the answers were about converting lists to numpy array. I think that's not the question in this error. My lists are all in numpy array format. This custom generator is based on a custom generator from here

This is the part of code where I fit the model:

        train_generator = RepresentationGenerator(representation_path=representations_path, target_path=target_path,
                                              filenames=training_filenames, batch_size=batch_size)
    val_generator = RepresentationGenerator(representation_path=representations_path, target_path=target_path,
                                            filenames=validation_filenames, batch_size=batch_size)
    self.model_semantic.fit_generator(train_generator,
                            epochs=10,
                            verbose=1,
                            validation_data=val_generator,
                            )
    return 0

where the variables are:

• representations_path - is a string with the directory to the path where i store the training files, that which file is the input to model
• target_path - is a string with the directory to the path where i store the target files, that which file is the target of the model (output)
• training_filenames - is a list with the names of training and target files (both have the same name, but they are in different folders)
• batch_size - integer with the size of the batch. It has the value 7.

My generator class is below:

import np
from tensorflow_core.python.keras.utils.data_utils import Sequence

class RepresentationGenerator(Sequence):

    def __init__(self, representation_path, target_path, filenames, batch_size):
        self.filenames = np.array(filenames)
        self.batch_size = batch_size
        self.representation_path = representation_path
        self.target_path = target_path

    def __len__(self):
        return (np.ceil(len(self.filenames) / float(self.batch_size))).astype(np.int)

    def __getitem__(self, idx):
        files_to_batch = self.filenames[idx * self.batch_size: (idx + 1) * self.batch_size]
        batch_x, batch_y = [], []
        for file in files_to_batch:
            batch_x.append(np.load(self.representation_path + file + ".npy", allow_pickle=True))
            batch_y.append(np.load(self.target_path + file + ".npy", allow_pickle=True))

        return np.array(batch_x), np.array(batch_y)

These are the values, when the method fit is called:

How can I fix this error?

Thank you mates!

---

When I call the method fit_generator, it calls the method fit.

The method fit, calls the method func.fit and it passes the variable Y that is set as None

The error occurs in this line:

Answer 1

Final solution:

Import from the correct place:

from tensorflow.keras.utils import Sequence

---

Old answers:

If __getitem__ is never called, the problem might be in __len__. You're not returning an int, you're returning a np.int.

I suggest you try:

def __len__(self):
    length = len(self.filenames) // self.batch_size
    if len(self.filenames) % self.batch_size > 0:
        length += 1

    return length

But if __getitem__ is being called and your data returned, then you should inspect your arrays.

Get an item from the generator yourself and check the content:

x, y = train_generator[0]

• Are they single arrays? Or are they arrays of arrays? (Must be single)
• What are their shapes? Do they have the expected shapes?
• What are their types? Usually they should be float, sometimes int (for inputs to embedding layers), very rarely string (for inputs to custom layers that know how to treat strings).
  • The outputs must always be float, at most int (for sparse losses)

---

Other suppositions, you're using fit with batch_size while using a generator.... this is strange and the "if" clauses inside the method may not be well prepared, you might be falling into another training case.

Go straight to the usual options:

self.model_semantic.fit_generator(train_generator, 
                                 epochs=10,
                                 verbose=1,
                                 validation_data=val_generator)

Your generator is a Sequence, it already has a __len__, you don't need to specify steps_per_epoch or validation_steps.
Every generator has automatic batch sizes, every step is a batch and that's it. You don't need to specify batch_size in fit_generator.

If you're going to use fit, go like this:

...fit(train_generator, steps_per_epoch = len(train_generator), 
       epochs = 10, verbose = 1,
       validation_data = val_generator, validation_steps = len(val_generator))  

---

Finally, you should be hunting for anything that might be None (as the error message suggests) in your code.

• Check if every function has a return line.
• Check all inputs of your generator in __init__.
• Print the filenames inside the generator.
• Get the __len__ of the generator.
• Try to get an item from the generator: x, y= train_generator[0]