CODEWITHSUNDEEP
pythonkerasneural-network
BotMan
BotMan

Reputation: 37

Proper use fit_generator in keras

All welcome. Trying to understand how fit_generator works in keras.

I have dataset, in each file - 100 images and 100 labels.

I wrote this generator:

def GenerateData(self):

    while True:

        complete_x1 = np.zeros((500, 50, 50, 3))
        complete_x2 = np.zeros((500, 50, 50, 3))
        complete_y1 = np.zeros((500, 3))
        complete_y2 = np.zeros((500, 2))

        done = 0

        while done < 500:

            data = np.load("{}/data_resized_{}.npy".format(self._patch, self._LastID))

            self.Log('\nLoad ALL data. ID: {} - Done: {}'.format(self._LastID, done))

            for data_x1, data_x2, data_y1, data_y2 in data:

                data_x1 = self.random_transform(data_x1)

                data_x2 = self.random_transform(data_x2)

                data_x1 = self.ImageProcessing(data_x1, 0)

                data_x2 = self.ImageProcessing(data_x2, 1)

                data_x1 = np.array(data_x1).astype('float32')
                data_x1 /= 255

                data_x2 = np.array(data_x2).astype('float32')
                data_x2 /= 255

                complete_x1[done] = data_x1
                complete_x2[done] = data_x2

                complete_y1[done] = data_y1
                complete_y2[done] = data_y2

                done += 1

            self._LastID += 1

            if self._LastID >= 1058:
                self._LastID = 0

        yield [np.array(complete_x1), np.array(complete_x2)], [np.array(complete_y1), np.array(complete_y2)]

I have 1058 files in total. It turns out 105800 images with labels.

Model training:

model.fit_generator(data.GenerateData(), samples_per_epoch=1058/500, nb_epoch=15, verbose=1, workers=1)

Everything seems to be good, but!

At the very beginning of training, GenerateData print the following:

Load ALL data. ID: 0 - Done: 0

Load ALL data. ID: 1 - Done: 100

Load ALL data. ID: 2 - Done: 200

Load ALL data. ID: 3 - Done: 300

Load ALL data. ID: 4 - Done: 400

Load ALL data. ID: 5 - Done: 0

Load ALL data. ID: 6 - Done: 100

Load ALL data. ID: 7 - Done: 200

Load ALL data. ID: 8 - Done: 300

Load ALL data. ID: 9 - Done: 400

Load ALL data. ID: 10 - Done: 0

And this happens before the file with ID 59. It turns out ... Does it skip everything that goes up to the 59 file? 5900 images?

It simply loads 500 images, after which it passes yield and starts again, with the ID of the file on which he finished, but the train does not work.

Here is what comes after the 59th file:

Load ALL data. ID: 59 - Done: 400 1/2 [=============>................] - ETA: 4s - loss: 2.8177 - dense_18_loss: 2.0145 - dense_21_loss: 0.8032 - dense_18_acc: 0.2140 - dense_21_acc: 0.5780 Load ALL data. ID: 60 - Done: 0

Load ALL data. ID: 61 - Done: 100

Load ALL data. ID: 62 - Done: 200

Load ALL data. ID: 63 - Done: 300

Load ALL data. ID: 64 - Done: 400 2/2 [===========================>..] - ETA: 0s - loss: 2.7260 - dense_18_loss: 1.7077 - dense_21_loss: 1.0183 - dense_18_acc: 0.2720 - dense_21_acc: 0.5890 Load ALL data. ID: 65 - Done: 0

Load ALL data. ID: 66 - Done: 100

Why is this happening?

Upvotes: 1

Views: 54

Answers (1)

Mitiku
Mitiku

Reputation: 5412

You are getting this behavior because you set workers to 1, and the data generating task and training task are running on separate threads. The training task runs on the main thread, while the data generating task runs on separate thread(s), where the number of threads depend on the workers argument.

If the workers argument had been 0, the data-generator would run on the main thread and the result would have been what you are expecting.

Upvotes: 1

Related Questions