CODEWITHSUNDEEP
tensorflowkeraspython-3.6numpy-ndarray
Js541
Js541

Reputation: 110

How to create training and testing data and feed into keras model?

It might be a very simple question, but as a newbie in keras and machine learning, I am unable to solve this issue. This is a two-class classification problem. My code is written (Source:Kaggle) in keras with a Tensorflow backend.

I have a directory that contains two folders named "cat" and "dog". Each folder has multiple images of size 224 x 224 pixels. The total image size is more than 32 GB. The label will be based on the folder names, i.e., if folder name contains "cat", the label will be "0" else "1".

Code Snippet (Source:Kaggle):

def get_images(directory):
    Images = []
    Labels = []  
    label = 0
    for labels in os.listdir(directory): #Main Directory where each class label is present as folder name.
        if labels == 'cat': #Folder contain 'cat' Images get the '0' class label.
            label = 0
        elif labels == 'dog':
            label = 1

        for image_file in os.listdir(directory+labels): #Extracting the file name of the image from Class Label folder
            image = cv2.imread(directory+labels+r'/'+image_file) #Reading the image (OpenCV)        
            image = cv2.resize(image,(224,224)) #Resize the image, Some images are different sizes. (Resizing is very Important)
            Images.append(image)
            Labels.append(label)

    return shuffle(Images,Labels,random_state=817328462) #Shuffle the dataset you just prepared. 817328462 

def get_classlabel(class_code):    
    labels = {0:'cat', 1:'dog'}
    return labels[class_code]

Images, Labels = get_images('./path_of_data_set') #Extract the training images from the folders.
Images = np.array(Images)
Labels = np.array(Labels)

def sequence():
    model = Models.Sequential()
    ...
model=sequence();
model.summary()

# Train the model with the new callback
model.fit(Images, Labels, batch_size=32, epochs=100, validation_split=0.10, verbose=1)

If the number of .png images is small, then my code is running perfectly. The problem arises when I am using 32GB image data. Then I am getting a memory issue. I check lots of post in this regard and found lots of solutions, but I am unable to implement them in this code.

Can you please tell me how can I feed the data into the model, so that it should not show memory issues?

Upvotes: 0

Views: 602

Answers (1)

user1410665
user1410665

Reputation: 769

Check here. Details are available. You may need to add few more lines. https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator

Upvotes: 1

Related Questions