CODEWITHSUNDEEP
pythontensorflowsplit
Cillin O Foghlu
Cillin O Foghlu

Reputation: 187

python.data.ops.dataset_ops.BatchDataset - how to use it to create training and test datasets

Using TensorFlow to walk directories and take images which i want to use in training a NN.

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    wk_dir,
    labels="inferred",
    label_mode="int",
    class_names=None,
    color_mode="grayscale",
    batch_size=batches,
    image_size=image_dim,
    shuffle=True,
    seed=1968,
    validation_split=0.2,
    subset="training",
    interpolation="bilinear",
    follow_links=False,
)

Found 127561 files belonging to 3 classes. Using 102049 files for training.

Result - it works....now i am trying to use this to input into a model and not sure how to manage it... 

print(train_ds)
<BatchDataset shapes: ((None, 576, 432, None), (None,)), types: (tf.float32, tf.int32)>

So do i have 2 elements in the array one with 4 elements, 2 of which are empty and a 2nd element whic his the classification?
i have tried to split the BatchDatashape and get error TypeError: 'BatchDataset' object is not subscriptable

how do you manipulate a TF of object type python.data.ops.dataset_ops.BatchDataset ?

Upvotes: 7

Views: 9340

Answers (2)

VaibhavPandey
VaibhavPandey

Reputation: 1

You have one dataset name -train_ds. If you want validation dataset you will need to write one more statement, difference is in the subset name:

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    wk_dir,
    labels="inferred",
    label_mode="int",
    class_names=None,
    color_mode="grayscale",
    batch_size=batches,
    image_size=image_dim,
    shuffle=True,
    seed=1968,
    validation_split=0.2,
    subset="validation",
    interpolation="bilinear",
    follow_links=False,
)

Upvotes: -1

Ena
Ena

Reputation: 126

  1. If you want to see how this BatchDataset looks like, you can try:

    print(list(train_ds.as_numpy_iterator()))

  2. More about TensorFlow Data and BatchDataset: https://www.tensorflow.org/guide/data#batching_dataset_elements

  3. Looks like there is no enough information to tell you how exactly build the model, but I can recommend this course to see how to build the model with BatchDataset as model input: https://www.coursera.org/projects/fine-tune-bert-tensorflow

Upvotes: 6

Related Questions