CODEWITHSUNDEEP
tensorflowneural-networktensorflow-datasets
AdamS
AdamS

Reputation: 11

VAalue_Error: as_list() is not defined on an unknown TensorShape when using dataset generator

There are a number of previous answers to this error but none relate to my issue.

I am creating a tensorflow dataset using a generator and for the examples sake am trying fit a very simple API model.

The dataset looks like this:

for example in ds.take(1):
    print(example[0], example[1])

Outputs:

{'LoB': <tf.Tensor: shape=(1,), dtype=int32, numpy=array([5], dtype=int32)>, 
'cc': <tf.Tensor: shape=(1,), dtype=int32, numpy=array([17], dtype=int32)>, 
'inj_part': <tf.Tensor: shape=(1,), dtype=int32, numpy=array([41], dtype=int32)>, 
'age': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([2.3495796], dtype=float32)>, 'RepDel': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([-0.26196158], dtype=float32)>, 
'dev_year_predictor': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([-1.2747549], dtype=float32)>, 
'cum_loss': <tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.8005615], dtype=float32)>} 
tf.Tensor([5.], shape=(1,), dtype=float32)

When I run this simple model:

inputs = layers.Input(shape=(1, ), name="LoB")
output = layers.Dense(1, activation="linear")(inputs)
test = models.Model(inputs=inputs, outputs=output)

test.compile(loss="mse", optimizer="sgd")

test.fit(ds, epochs=5, verbose=True)

I get the error: ValueError: as_list() is not defined on an unknown TensorShape

I managed to fix this by reshaping the tensors to shape=(1,), but I changed the way the dataset generator functions and it broke again. I cannot see any tensors in my dataset with unknown shape hence I am confused.

The Dataset was constructed from tabular data like this (simplified version but I still get the error):

def create_tensor(seq):

    LoB=seq["LoB"].values[0]
    target=seq[target].values[-1]
    return {"LoB": LoB}, target

def pad_and_format(seq):
    
    x, y = seq
    x["LoB"] = tf.reshape(x["LoB"], shape=(1,))
    y = tf.reshape(y, shape=(1,))
    return x,y

def generator():
    
    for i in range(train_df["ClNr_sub"].max()+1):
        seq=train_df[train_df["ClNr_sub"] == i]
        seq=create_tensor(seq)
        seq=pad_and_format(seq)
        yield seq

ds = tf.data.Dataset.from_generator(generator, 
                        output_types=({'LoB': tf.int32}, tf.float32))

Upvotes: 0

Views: 46

Answers (1)

Sagar
Sagar

Reputation: 169

The error arises because tensorflow cannot determine the tensor shapes explicitly. When using tf.data.Dataset.from_generator, you should provide an output_signature argument to explicitly define the shape and type of the output tensors. This allows tensorflow to properly handle the data.

Instead of using this:

ds = tf.data.Dataset.from_generator(generator, 
                        output_types=({'LoB': tf.int32}, tf.float32))

Use this:

ds = tf.data.Dataset.from_generator(generator, output_signature=(
    {'LoB': tf.TensorSpec(shape=(1,), dtype=tf.int32)},
    tf.TensorSpec(shape=(1,), dtype=tf.float32)
))

please refer to this document for more details.

Upvotes: 0

Related Questions