CODEWITHSUNDEEP
pythontensorflow
Aleksei Solovev
Aleksei Solovev

Reputation: 59

Sample a sequence of images from a folder with TensorFlow and tf.data.Dataset

Consider a data frame with folder names and corresponding labels. Each folder contains an arbitrary number of images from video files. I'm looking for a way to sample a sequence of images from a folders with tf.data.Dataset to train an action recognition model. Something like that:

ds = tf.data.Dataset.from_tensor_slices(list_of_folders)

def read_and_preprocess_images_from_folder(folder):
    list_of_image_names = some_function_to_list_files(folder)
    list_length = len(list_of_image_names)

    upper_boundary = list_length - sequence_length
    random_start_index = tf.random_uniform(shape=[], minval=0, maxval=upper_boundary, dtype=tf.int64)

    random_sequence = list_of_image_names[random_start_index:random_start_index+sequence_length]

    return convert_sequence_to_image_tensor(random_sequence)

What I've done so far:

df = pd.DataFrame({'folder': ['folder_0', 'folder_1'], 'target': [0, 1]})

ds = tf.data.Dataset.from_tensor_slices((df.folder.values, df.target.values))

def load_and_preprocess_image_sequence(folder):
    x = tf.io.matching_files('/path/to/folders/' + folder + '/*.jpg') 
    x = tf.map_fn(lambda x: preprocess_image(tf.read_file(x)), x, dtype=tf.float32)
    return x

def preprocess_image(x):
    x = tf.image.decode_jpeg(x, channels=3)
    x = tf.image.resize_images(x, size=(IMAGE_SIZE,IMAGE_SIZE))
    return x

def load_and_preprocess_from_folder_label(folder, label):
    return load_and_preprocess_image_sequence(folder), label

train_ds = train_ds.map(load_and_preprocess_from_folder_label)

And I get:

<DatasetV1Adapter shapes: ((?, 224, 224, 3), ()), types: (tf.float32, tf.int64)>

The problem is that tf.io.matching_files returns a tensor with no shape when used with tf.data.Dataset. It returns a defined shape only during eager execution.

I tried to solve this problem differently. Knowing that every image in every folder has the same structure ['0001.jpg', '0002.jpg'] I tried to use np.random.randint but the problem is that np.random.randint produces the same result every time:

def load_and_preprocess_image_sequence(folder):
    random_start_index = np.random.randint(0,upper_boundary) 
    x = []
    for i in range(random_start_index, random_start_index+sequence_length):
        x.append('/path/to/folders/' + folder + f'/{i:04d}.jpg')

    x = [tf.read_file(i) for i in x]
    x = [preprocess_image(i) for i in x]    
    x = tf.stack(x, axis=0)
    return x

It works fine except the same random_start_index every time. In order to solve the randomness issue I have to use tf.random_uniform:

def load_and_preprocess_image_sequence(folder):
    random_start_index = tf.random_uniform(shape=[], minval=0, maxval=upper_boundary, dtype=tf.int64)
    range = tf.map_fn(lambda x: x + random_start_index, tf.range(sequence_length, dtype=tf.int64))

And I get a tensor of consecutive numbers starting at random with the length equals to the sequence_length. The problem now is that tf.strings.format is somewhat limited and cannot produce results on a par with python formatting, like for example f'{i:04d}.jpg'.

Upvotes: 2

Views: 850

Answers (1)

Aleksei Solovev
Aleksei Solovev

Reputation: 59

I was able to solve this. Here is an example:

x = tf.io.matching_files(folder + '/*.jpg')
max_start_index = tf.cast(len(x) - SEQUENCE_LEN, tf.int64)

if max_start_index == 0:
    random_start_index = max_start_index
else:
    random_start_index = tf.random.uniform(shape=[], minval=0, maxval=max_start_index, dtype=tf.int64)

x = x[random_start_index:random_start_index + SEQUENCE_LEN]
x = tf.map_fn(lambda x: load_image(x), x, dtype=tf.uint8)

Upvotes: 2

Related Questions