CODEWITHSUNDEEP
pythontensorflowmachine-learningkerasdeep-learning
eng2019
eng2019

Reputation: 1035

Using generator in Python to feed into Keras model.fit_generator

I am learning how to use generator in Python and feed it into Keras model.fit_generator.

from keras.utils import to_categorical
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Flatten
import pandas as pd
import os
import cv2

class Generator:
    def __init__(self,path):
        self.path = path

    def gen(self, feat, labels):
        i=0
        while (True):
            im = cv2.imread(feat[i],0)
            im = im.reshape(28,28,1)
            yield im,labels[i]
            i+=1

if __name__ == "__main__":
    input_dir = './mnist'
    output_file = 'dataset.csv'

    filename = []
    label = []
    for root,dirs,files in os.walk(input_dir):
        for file in files:
            full_path = os.path.join(root,file)
            filename.append(full_path)
            label.append(os.path.basename(os.path.dirname(full_path)))

    data = pd.DataFrame(data={'filename': filename, 'label':label})
    data.to_csv(output_file,index=False)

    feat = data.iloc[:,0]
    labels = pd.get_dummies(data.iloc[:,1]).as_matrix()

    image_gen = Generator(input_dir)

    # #create model
    model = Sequential()
    model.add(Conv2D(64, kernel_size=3, activation="relu", input_shape=(28,28,1)))
    model.add(Conv2D(32, kernel_size=3, activation="relu"))
    model.add(Flatten())
    model.add(Dense(2, activation="softmax"))

    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    model.fit_generator(image_gen.gen(filename,labels), steps_per_epoch=5 ,epochs=5, verbose=1)

I have 2 subfolders inside ./mnist folder, corresponding to each class in my dataset. I created a Dataframe that contains the path of each image and the label (which is the name of the corresponding subfolder).

I created Generator class that loads the image whose path is written in the DataFrame.

It gave me error: ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (28, 28, 1)

Could anyone please help? And also, is it the correct way to implement generator in general?

Thanks!

Upvotes: 0

Views: 275

Answers (1)

roman_ka
roman_ka

Reputation: 578

I think answers to your questions can be found in Keras documentation.

In terms of the input shape, Conv2D layers expects 4-dimensional input, but you explicitly reshape to (28,28,1) in your generator, so 3 dimensions. On the Conv2D info and the input format, see this documentation.

In terms of the generator itself, Keras documentation provides an example with generator being a function, the same is discussed in Python Wiki. But your particular implementation seem to work, at least for the first iteration, if you get to the point of feeding the data into the convolution layer.

Upvotes: 1

Related Questions