Reputation: 110
It might be a very simple question, but as a newbie in keras and machine learning, I am unable to solve this issue. This is a two-class classification problem. My code is written (Source:Kaggle) in keras with a Tensorflow backend.
I have a directory that contains two folders named "cat" and "dog". Each folder has multiple images of size 224 x 224 pixels
. The total image size is more than 32 GB. The label will be based on the folder names, i.e., if folder name contains "cat", the label will be "0" else "1".
Code Snippet (Source:Kaggle):
def get_images(directory):
Images = []
Labels = []
label = 0
for labels in os.listdir(directory): #Main Directory where each class label is present as folder name.
if labels == 'cat': #Folder contain 'cat' Images get the '0' class label.
label = 0
elif labels == 'dog':
label = 1
for image_file in os.listdir(directory+labels): #Extracting the file name of the image from Class Label folder
image = cv2.imread(directory+labels+r'/'+image_file) #Reading the image (OpenCV)
image = cv2.resize(image,(224,224)) #Resize the image, Some images are different sizes. (Resizing is very Important)
Images.append(image)
Labels.append(label)
return shuffle(Images,Labels,random_state=817328462) #Shuffle the dataset you just prepared. 817328462
def get_classlabel(class_code):
labels = {0:'cat', 1:'dog'}
return labels[class_code]
Images, Labels = get_images('./path_of_data_set') #Extract the training images from the folders.
Images = np.array(Images)
Labels = np.array(Labels)
def sequence():
model = Models.Sequential()
...
model=sequence();
model.summary()
# Train the model with the new callback
model.fit(Images, Labels, batch_size=32, epochs=100, validation_split=0.10, verbose=1)
If the number of .png images is small, then my code is running perfectly. The problem arises when I am using 32GB image data. Then I am getting a memory issue. I check lots of post in this regard and found lots of solutions, but I am unable to implement them in this code.
Can you please tell me how can I feed the data into the model, so that it should not show memory issues?
Upvotes: 0
Views: 602
Reputation: 769
Check here. Details are available. You may need to add few more lines. https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator
Upvotes: 1