CODEWITHSUNDEEP
pythonnumpytensorflowtuplestensorflow-datasets
Dingsda
Dingsda

Reputation: 11

Problem with Tensorflow iterator returning tuples

I want to iterate over a TF dataset in order to convert the obtained data to numpy tensors. Being new to tensorflow, this is what my code looks like

  def convert_dataset_to_pytorch(self, dataset):
    sess = tf.Session(config=self.config)

    iterator = dataset.make_one_shot_iterator()
    exampleTF, labelsTF = iterator.get_next()

    examples = torch.Tensor()
    labels = torch.Tensor()

    try:
      while True:
        examples = torch.cat((examples,torch.Tensor(exampleTF.eval(session=sess))),0)
        labels = torch.cat((labels,torch.Tensor([labelsTF.eval(session=sess)])),0)
    except tf.errors.OutOfRangeError:
      pass

    return examples, labels

The apparent problem is that every call to eval() iterates both over exampleTF and labelsTF, thus skipping half of the entries. Any help? I also tried something like

  def convert_dataset_to_pytorch(self, dataset):
    sess = tf.Session(config=self.config)

    iterator = dataset.make_one_shot_iterator()
    next_element = iterator.get_next()

    examples = torch.Tensor()
    labels = torch.Tensor()

    try:
      while True:
        sess.run(next_element)
        examples = torch.cat((examples,torch.Tensor(next_element[0])),0)
        labels = torch.cat((labels,torch.Tensor([next_element[0]])),0)
    except tf.errors.OutOfRangeError:
      pass

    return examples, labels

but this results only in errors of the form

examples = torch.cat((examples,torch.Tensor(next_element[0])),0)
TypeError: object of type 'Tensor' has no len()

Upvotes: 0

Views: 536

Answers (1)

mujjiga
mujjiga

Reputation: 16906

Not sure why you are creating a pytorch tensor in tensorflow when all you want is a numpy tensor. To answer your question (mentioned below)

iterate over a TF dataset in order to convert the obtained data to numpy tensors.

Sample Code:

import numpy as np

inc_dataset = tf.data.Dataset.range(100)
dec_dataset = tf.data.Dataset.range(0, -100, -1)
dataset = tf.data.Dataset.zip((inc_dataset, dec_dataset))

iterator = dataset.make_one_shot_iterator()
next_element = iterator.get_next()

result = list()
with tf.Session() as sess:
    try:
        while True:
          result.append(sess.run(next_element)) 
    except tf.errors.OutOfRangeError:
          pass

examples = np.array(list(zip(*result))[0])
labels = np.array(list(zip(*result))[1])

Now you can convert examples and labels np arrays to pytorch or tensorflow tensors or to whatever tensors you want.

Upvotes: 2

Related Questions