CODEWITHSUNDEEP
pythontensorflowtensorflow-datasets
Nasty
Nasty

Reputation: 131

Using tf.data.Dataset.from_generator() with a function (tensorflow or numpy) as the generating source (instead of a file)

The script is attempting to use a function (np.sin() in this case) as a generator designed to ultimately be pipelined into model for training (hence the tf.Session()). Unfortunately, I keep getting the error:

"ValueError: generator yielded an element of shape () where an element of shape (1,) was expected."

I've used generators in the past to pull data from .hdf5 files, but what I'm trying to do here is instead generate waveform data from a function within the callable class.

Calling the generating function outside of the tf.data.Dataset.from_generator() works as desired:

next(sine_wave_source())

import numpy as np
import tensorflow as tf

class sine_wave_source:

    def __init__(self,frequency = 1,sampling_frequency = 100):
        self.fc = frequency
        self.Fs = sampling_frequency
        self.time_vector = np.arange(0,1,1/self.Fs,dtype = 'float32')

    def __call__(self):
        for t in self.time_vector:
            yield np.sin(2*np.pi*self.fc*t,dtype = 'float32')

data_gen = tf.data.Dataset.from_generator(
            sine_wave_source(),
            output_types = (tf.float32),
            output_shapes = (tf.TensorShape([1])))

data_iterator = data_gen.make_initializable_iterator()

next_sample = data_iterator.get_next()

with tf.Session() as sess:
        sess.run(data_iterator.initializer)
        for ii in range(0,100):
           sample = sess.run([next_sample])
           print(sample)

Upvotes: 2

Views: 4588

Answers (1)

javidcf
javidcf

Reputation: 59701

With output_shapes = (tf.TensorShape([1])) you are indicating that each item in the dataset will be a one-dimensional tensor with one element. However, the generator in sine_wave_source returns scalar values. TensorFlow is strict here, it could just broadcast to the one-item vector, but considers it an error. You can either change output_shapes like:

data_gen = tf.data.Dataset.from_generator(sine_wave_source(),
                                          output_types=tf.float32,
                                          output_shapes=tf.TensorShape([]))

Or change the generator to return a list or an array, e.g.:

def __call__(self):
    for t in self.time_vector:
        yield [np.sin(2 * np.pi * self.fc * t, dtype='float32')]

Upvotes: 3

Related Questions