tf.data generates training results that do not improve

Question

I am trying to use tf.data to augment a dataset that I have. The dataset is arranged locally in my computer like this:

datasets/fruits/{class_name}/*jpg

{class_name} include 7 different kinds of fruits, including: strawberry, mango, broccoli, grape, apple, lemon, and orange.

This is the code I have written for the data augmentation step. As you can see, I'm not even actually augmenting the data, I'm only loading the images using tf.data.Dataset.from_tensor_slices and rescaling the pixels:

import tensorflow as tf
import random
from tensorflow.data import AUTOTUNE
from tensforflow.keras.optimizers import SGD
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.layers.experimental import preprocessing
from imutils.import paths
from sklearn.preprocessing import LabelEncoder

INIT_LR = 1e-2 # learning rate
BS = 32 # batch size
EPOCHS = 50 # number of epochs

# load images with tensorflow

def load_images(imagePath, label):
    image = tf.io.read_file(imagePath)
    image = tf.image.decode_jpeg(image, channels=3)
    image = tf.image.convert_image_dtype(image, dtype=tf.float32)
    image = tf.image.resize(image, (64, 64))
    return (image, label)

# augment helper function
def augment(image, label, aug):
    image = aug(image)

    return (image, label)

# get all image paths and save them as strings with format **/{class_name}/*jpg
allImages = list(paths.list_images("datasets/fruits"))
random.shuffle(allImages) # shuffle the images

# perform 0.75/0.25 train/test split
i = int(len(allImages) * 0.25)
trainPaths = allImages[i:]

# get labels by getting {class_name} from **/{class_name}/*jpg
trainLabels = [p.split(os.path.sep)[-2] for p in trainPaths]
testPaths = allImages[:i]
testLabels = [p.split(os.path.sep)[-2] for p in testPaths]

# use LabelEncoder to one-hot encode the class names
labelEncoder = LabelEncoder()
labelEncoder = labelEncoder.fit(trainLabels)
trainLabels = labelEncoder.transform(trainLabels)
trainLabels = to_categorical(trainLabels)
testLabels = labelEncoder.transform(testLabels)
testLabels = to_categorical(testLabels)

# load the train and test data into a tf.data.Dataset
trainDS = tf.data.Dataset.from_tensor_slices((trainPaths, trainLabels))
trainDS = (
    trainDS
    .shuffle(32, seed=42)
    .map(load_images, num_parallel_calls=AUTOTUNE)
    .batch(BS)
    .cache()
)

# rescale the pixels from [0, 1]

trainAug = tf.keras.Sequential(
    [
        preprocessing.Rescaling(scale=1.0/255),
    ]
)

trainDS = (
    trainDS
    .map(lambda x, y: augment(x, y, trainAug), num_parallel_calls=AUTOTUNE)
    .prefetch(AUTOTUNE)
)

testDS = tf.data.Dataset.from_tensor_slices(( testPaths, testLabels ))
testDS = (
    testDS
    .shuffle(32)
    .map(load_images, num_parallel_calls=AUTOTUNE)
    .batch(BS)
    .cache()
)

testAug = tf.keras.Sequential(
    [
        preprocessing.Rescaling(scale=1.0/255),
    ]
)

testDS = (
    testDS
    .map(lambda x, y: augment(x, y, testAug), num_parallel_calls=AUTOTUNE)
    .prefetch(AUTOTUNE)
)

# I don't think there is any issues with this part, but here I am setting up the optimizer and model for training
sgd = SGD(learning_rate=INIT_LR, momentum=0.9, weight_decay=INIT_LR/EPOCHS)
model = MiniVGGNet.build(64, 64, 3, num_classes=num_classes)
model.compile(loss="categorical_crossentropy", optimizer=sgd, metrics=["accuracy"])
training_history = model.fit(
    x=trainDS,
    validation_data=testDS,
    epochs=EPOCHS
)

Here are the training results I am getting. You can see the val_loss and val_accuracy show no sign of improving. training results using tf.data to load images

I suspect the issue is with the way image data is loaded and processed and not with the model I am using. I was able to generate reasonable results using the same model, training on the images loaded not with tensorflow methods, but simple python methods. These were the results I got training results using standard python methods to load data

I would expect to get similar training results, regardless of how I am loading my image data. If anyone can shed any light on my issue, I would greatly appreciate it! I've been stuck on this for several days now.

Tried using tf.data to load the dataset and train, but got strange results

Answer 1

Turns out I was rescaling the pixels twice.

image = tf.image.convert_image_dtype(image, dtype=tf.float32)

rescales automatically once, and later, in the preprocessing step, I rescaled again

trainAug = tf.keras.Sequential(
    [
        preprocessing.Rescaling(scale=1.0/255),
    ]
)

after removing preprocessing.Rescaling(scale=1.0/255),, the training started to generate sensible results. The documentation mentions this auto-rescaling behavior (https://www.tensorflow.org/api_docs/python/tf/image/convert_image_dtype)