Reputation: 487
i'm trying to fit my deep learning model with a custom generator.
When i fit the model, it shows me this error:
I tried to find similar questions, but all the answers were about converting lists to numpy array. I think that's not the question in this error. My lists are all in numpy array format. This custom generator is based on a custom generator from here
This is the part of code where I fit the model:
train_generator = RepresentationGenerator(representation_path=representations_path, target_path=target_path,
filenames=training_filenames, batch_size=batch_size)
val_generator = RepresentationGenerator(representation_path=representations_path, target_path=target_path,
filenames=validation_filenames, batch_size=batch_size)
self.model_semantic.fit_generator(train_generator,
epochs=10,
verbose=1,
validation_data=val_generator,
)
return 0
where the variables are:
My generator class is below:
import np
from tensorflow_core.python.keras.utils.data_utils import Sequence
class RepresentationGenerator(Sequence):
def __init__(self, representation_path, target_path, filenames, batch_size):
self.filenames = np.array(filenames)
self.batch_size = batch_size
self.representation_path = representation_path
self.target_path = target_path
def __len__(self):
return (np.ceil(len(self.filenames) / float(self.batch_size))).astype(np.int)
def __getitem__(self, idx):
files_to_batch = self.filenames[idx * self.batch_size: (idx + 1) * self.batch_size]
batch_x, batch_y = [], []
for file in files_to_batch:
batch_x.append(np.load(self.representation_path + file + ".npy", allow_pickle=True))
batch_y.append(np.load(self.target_path + file + ".npy", allow_pickle=True))
return np.array(batch_x), np.array(batch_y)
These are the values, when the method fit is called:
How can I fix this error?
Thank you mates!
When I call the method fit_generator, it calls the method fit.
The method fit, calls the method func.fit and it passes the variable Y that is set as None
The error occurs in this line:
Upvotes: 2
Views: 2742
Reputation: 86650
Final solution:
Import from the correct place:
from tensorflow.keras.utils import Sequence
Old answers:
If __getitem__
is never called, the problem might be in __len__
. You're not returning an int
, you're returning a np.int
.
I suggest you try:
def __len__(self):
length = len(self.filenames) // self.batch_size
if len(self.filenames) % self.batch_size > 0:
length += 1
return length
But if __getitem__
is being called and your data returned, then you should inspect your arrays.
Get an item from the generator yourself and check the content:
x, y = train_generator[0]
float
, sometimes int
(for inputs to embedding layers), very rarely string
(for inputs to custom layers that know how to treat strings).
float
, at most int
(for sparse losses) Other suppositions, you're using fit
with batch_size
while using a generator.... this is strange and the "if" clauses inside the method may not be well prepared, you might be falling into another training case.
Go straight to the usual options:
self.model_semantic.fit_generator(train_generator,
epochs=10,
verbose=1,
validation_data=val_generator)
Your generator is a Sequence
, it already has a __len__
, you don't need to specify steps_per_epoch
or validation_steps
.
Every generator has automatic batch sizes, every step is a batch and that's it. You don't need to specify batch_size
in fit_generator
.
If you're going to use fit
, go like this:
...fit(train_generator, steps_per_epoch = len(train_generator),
epochs = 10, verbose = 1,
validation_data = val_generator, validation_steps = len(val_generator))
Finally, you should be hunting for anything that might be None
(as the error message suggests) in your code.
return
line. __init__
. __len__
of the generator. x, y= train_generator[0]
Upvotes: 1