RNN Model Error: "ValueError: This model has not yet been built."

Question

I'm building a character-based LSTM-RNN text generator using this tutorial on Google Colab: https://colab.research.google.com/github/tensorflow/text/blob/master/docs/tutorials/text_generation.ipynb#scrollTo=d4tSNwymzf-q.

While their code runs and compiles on my Google Colab account with their Shakespeare dataset, it does not work when I input my own dataset. This error continuously comes up:

"ValueError: This model has not yet been built.

The dataset they used was the Shakespeare text from Tensorflow (https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt). My dataset, on the other hand, is in the form of short characters. Here are the first five lines of my dataset (I'm experimenting with generating peptide sequences):

acssspskhcg

agcknffwktftsc

agilkrw

agyllgkinlkalaalakkil

aplepeypgdnatpeqmaqyaaelrryinmltrpry

cagalcy

I'm thinking this might be part of the problem.

Here is the code that I have so far:

import tensorflow as tf
from tensorflow.keras.layers.experimental import preprocessing

import numpy as np
import os
import time

# Read, then decode for py2 compat.
text = open("/content/generatorinput.txt", 'rb').read().decode(encoding='utf-8')
# length of text is the number of characters in it
print(f'Length of text: {len(text)} characters')

# The unique characters in the file
vocab = sorted(set(text))
print(f'{len(vocab)} unique characters')

example_texts = ['acdefgh', 'tvy']
chars = tf.strings.unicode_split(example_texts, input_enco
chars

ids_from_chars = preprocessing.StringLookup(
    vocabulary=list(vocab), mask_token=None)

ids = ids_from_chars(chars)
ids

chars_from_ids = tf.keras.layers.experimental.preprocessing.StringLookup(
    vocabulary=ids_from_chars.get_vocabulary(), invert=True, mask_token=None)

chars = chars_from_ids(ids)
chars

tf.strings.reduce_join(chars, axis=-1).numpy()

def text_from_ids(ids):
  return tf.strings.reduce_join(chars_from_ids(ids), axis=-1)

all_ids = ids_from_chars(tf.strings.unicode_split(text, 'UTF-8'))
all_ids

ids_dataset = tf.data.Dataset.from_tensor_slices(all_ids)

for ids in ids_dataset.take(10):
    print(chars_from_ids(ids).numpy().decode('utf-8'))

seq_length = 100
examples_per_epoch = len(text)//(seq_length+1)

sequences = ids_dataset.batch(seq_length+1, drop_remainder=True)

for seq in sequences.take(1):
  print(chars_from_ids(seq))

def split_input_target(sequence):
    input_text = sequence[:-1]
    target_text = sequence[1:]
    return input_text, target_text

dataset = sequences.map(split_input_target)

for input_example, target_example in dataset.take(1):
    print("Input :", text_from_ids(input_example).numpy())
    print("Target:", text_from_ids(target_example).numpy())

# Batch size
BATCH_SIZE = 64

# Buffer size to shuffle the dataset
# (TF data is designed to work with possibly infinite sequences,
# so it doesn't attempt to shuffle the entire sequence in memory. Instead,
# it maintains a buffer in which it shuffles elements).
BUFFER_SIZE = 100

dataset = (
    dataset
    .shuffle(BUFFER_SIZE)
    .batch(BATCH_SIZE, drop_remainder=True)
    .prefetch(tf.data.experimental.AUTOTUNE))

dataset

# Length of the vocabulary in chars
vocab_size = len(vocab)

# The embedding dimension
embedding_dim = 256

# Number of RNN units
rnn_units = 1024

class MyModel(tf.keras.Model):
  def __init__(self, vocab_size, embedding_dim, rnn_units):
    super().__init__(self)
    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)
    self.gru = tf.keras.layers.GRU(rnn_units,
                                   return_sequences=True,
                                   return_state=True)
    self.dense = tf.keras.layers.Dense(vocab_size)

  def call(self, inputs, states=None, return_state=False, training=False):
    x = inputs
    x = self.embedding(x, training=training)
    if states is None:
      states = self.gru.get_initial_state(x)
    x, states = self.gru(x, initial_state=states, training=training)
    x = self.dense(x, training=training)

    if return_state:
      return x, states
    else:
      return x

model = MyModel(
    # Be sure the vocabulary size matches the `StringLookup` layers.
    vocab_size=len(ids_from_chars.get_vocabulary()),
    embedding_dim=embedding_dim,
    rnn_units=rnn_units)

for input_example_batch, target_example_batch in dataset.take(1):
    example_batch_predictions = model(input_example_batch)
    print(example_batch_predictions.shape, "# (batch_size, sequence_length, vocab_size)")

model.summary() # <-- This is where the code stops working 

What I've tried: Doing a restart of my runtime, changing my buffer size and defining an input shape.

When I define the input shape and go on with the code, I get this:

sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)
sampled_indices = tf.squeeze(sampled_indices, axis=-1).numpy()

ERROR: example_batch_predictions is not defined

Either way, I get an error. How do I fix this problem? Any advice is deeply appreciated.

Answer 1

If you try to pass some data to your model as you are trying to do with this line: example_batch_predictions = model(input_example_batch) (in your for loop), your model's summary would work, but notice how nothing gets printed inside your loop. The problem is you are using example_texts, which contains two strings and you are still using a batch_size of 64 and a sequence_length of 100. If you change your batch_size to say 2 and your sequence_length to 5, you should see an output like this:

Length of text: 100 characters
20 unique characters
a
c
s
s
s
p
s
k
h
c
tf.Tensor([b'a' b'c' b's'], shape=(3,), dtype=string)
Input : b'ac'
Target: b'cs'
(1, 2, 21) # (batch_size, sequence_length, vocab_size)
Model: "my_model_13"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_13 (Embedding)     multiple                  5376      
_________________________________________________________________
gru_13 (GRU)                 multiple                  3938304   
_________________________________________________________________
dense_13 (Dense)             multiple                  21525     
=================================================================
Total params: 3,965,205
Trainable params: 3,965,205
Non-trainable params: 0
_________________________________________________________________