CODEWITHSUNDEEP
pythontensorflow
ozoubia
ozoubia

Reputation: 15

Best method to load a large dataset in CNN using tensoflow

Good evening, I'm working on a Cat-Dog classification in CNN using tensorflow 1.4 and i want to know what is the best method to load the data into the CNN without having insufficient memory problems, knowing that the dataset is of size (8000 image for training, and 2000 image for testing).
I have tried loading the data by reading it in opencv and converting it into numpy array and using that numpy array for training and testing but this method seems to have memory problems (the code can be seen below). I have also tried reducing the batch size but it doesn't seem to work, is there a way of loading the data from disc? so could you please guide me into a clear solution with an implementation or a code. Here is the code that I used for loading the data:
this is a the function that labels the data

# function that labels the data
def label_img(img):
    word_label = img.split('.')[-3]
    # DIY One hot encoder
    if word_label == 'cat':
        return [1, 0]
    elif word_label == 'dog':
        return [0, 1]


and here is a function that creates the train data, the same function is used for test data.

def create_train_data():
    # Creating an empty list where we should store the training data
    # after a little preprocessing of the data
    training_data = []

    # tqdm is only used for interactive loading
    # loading the training data
    for img in tqdm(os.listdir(TRAIN_DIR)):
        # labeling the images
        label = label_img(img)

        path = os.path.join(TRAIN_DIR, img)

        # loading the image from the path and then converting them into
        # greyscale for easier covnet prob
        img = cv2.imread(path)
        img = cv2.resize(img, (256, 256))

        training_data.append([np.array(img), np.array(label)])

        # shuffling of the training data to preserve the random state of our data
    random.seed(101)
    random.shuffle(training_data)

    return training_data


here is the session code containing the batched images
here the TrainX and TrainY are just nparrays containing the images and labels

with tf.Session() as sess:
    sess.run(init)
    train_loss = []
    test_loss = []
    train_accuracy = []
    test_accuracy = []
    for i in range(training_iters):
        for batch in range(len(trainX)//batch_size):
            batch_x = trainX[batch*batch_size:min((batch+1)*batch_size, len(trainX))]
            batch_y = trainY[batch*batch_size:min((batch+1)*batch_size, len(trainY))]
            # Run optimization op (backprop).
            # Calculate batch loss and accuracy
            opt = sess.run(optimizer, feed_dict={x: batch_x,
                                                              y: batch_y})
            loss, acc = sess.run([cost, accuracy], feed_dict={x: batch_x,
                                                              y: batch_y})
        print("Iter " + str(i) + ", Loss= " + \
                      "{:.6f}".format(loss) + ", Training Accuracy= " + \
                      "{:.5f}".format(acc))
        print("Optimization Finished!")

        # Calculate accuracy for all 10000 mnist test images
        test_acc,valid_loss = sess.run([accuracy, cost], feed_dict={x: testX, y : testY})
        train_loss.append(loss)
        test_loss.append(valid_loss)
        train_accuracy.append(acc)
        test_accuracy.append(test_acc)
        print("Testing Accuracy:", "{:.5f}".format(test_acc))

Upvotes: 0

Views: 495

Answers (1)

velociraptor11
velociraptor11

Reputation: 604

Loading thousands of images into memory is not the right way to do it.

Hence, that's why Tensorflow has built-in APIs for loading the proper data. You can find more information for loading images here: https://www.tensorflow.org/tutorials/load_data/images#load_using_tfdata

Example taken from the link:

  1. Loading the training data generator, which returns an Iterator
train_data_gen = image_generator.flow_from_directory(directory=str(data_dir),
                                                     batch_size=BATCH_SIZE,
                                                     shuffle=True,
                                                     target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                     classes = list(CLASS_NAMES))
  1. Get training batch
image_batch, label_batch = next(train_data_gen)

Using this idea, you can make changes to your own code

Upvotes: 1

Related Questions