tam63
tam63

Reputation: 313

Resizing images in data preprocessing for training convolution network

I am trying to load data from jpeg files to train a convolution network. The images are large, with 24 million pixels however, so loading and using the full resolution is not practical.

To get the images to a more useful format I am trying to load each image, rescale it and then append it to a list. Once this is done, I can then convert the list into a numpy array and feed into the network for training as usual.

My problem is that my data set is very large and it takes about a second to rescale every image, which means it is not feasible to resize every image the way I have currently implemented this:

length_training_DF = 30000
for i in range(length_training_DF):
    im = plt.imread(TRAIN_IM_DIR + trainDF.iloc[i]['image_name'] + '.jpg')
    image = block_reduce(im, block_size=(10, 10, 1), func=np.max)
    trainX.append(image)

I have also used the following:

length_training_DF = 30000
from keras.preprocessing import image
for i in range(50):
    img = image.load_img(TRAIN_IM_DIR + trainDF.iloc[0]['image_name'] + '.jpg', target_size=(224, 224))
    trainX.append(ima)

Is there any way to load these images more quickly into a format for training a network? I have thought about using a keras dataset, perhaps by using tf.keras.preprocessing.image_dataset_from_directory(), but the directory in which the image data is stored is not formatted correctly into folders containing the same targets as is required by this method.

The images are for a binary classification problem.

Upvotes: 1

Views: 959

Answers (1)

Lukasz Tracewski
Lukasz Tracewski

Reputation: 11377

The usual way would be to write a preprocessing script that loads the large images, rescales them, applies other operations if needed, and then saves each class to a separate directory, as required by ImageDataGenerator.

There are at least three good reasons to do that:

  • Typically, you will run your training process dozens of time. You don't want to every time do the rescaling or e.g. auto-white balance.
  • ImageDataGenerator provides vital methods for augmenting your training data set.
  • It's a good generator out of the box. Likely you don't want to load entire data set into memory.

Upvotes: 2

Related Questions