CODEWITHSUNDEEP
apache-sparktensorflowmnist
Mahsa Pourjafarian
Mahsa Pourjafarian

Reputation: 35

Error in the first step of running this example: TensorFlowOnSpark on a Spark Standalone cluster

I have a problem with running this example TensorFlowOnSpark on a Spark Standalone cluster (Single Host):

After executing mnist_data_setup.py file, it extracts the MNIST zip files correctly. But by calling extract_images(filename) functions, it faces an error. Please see the error in the following:

 Extracting <open file 'FILE_PATH_IN_MT_PC/train-images-idx3-ubyte.gz', mode 'rb' at 0x7ff3423e5c00>    
Traceback (most recent call last):
  File "FILE_PATH_IN_MT_PC/mnist/mnist_data_setup.py", line 144, in <module>
    writeMNIST(sc, "FILE_PATH_IN_MT_PC/train-images-idx3-ubyte.gz", "FILE_PATH_IN_MT_PC/train-labels-idx1-ubyte.gz", args.output + "/train", args.format, args.num_partitions)
  File "/FILE_PATH_IN_MT_PC/mnist/mnist_data_setup.py", line 52, in writeMNIST
    images = numpy.array(mnist.extract_images(f))
  File "FILE_PATH_IN_MT_PC/tensorflow/contrib/learn/python/learn/datasets/mnist.py", line 42, in extract_images
    with tf.gfile.Open(filename, 'rb') as f, gzip.GzipFile(fileobj=f) as bytestream:
  File "FILE_PATH_IN_MT_PC/tensorflow/python/platform/gfile.py", line 452, in Open
    return GFile(name, mode=mode)
  File "FILE_PATH_IN_MT_PC/tensorflow/python/platform/gfile.py", line 215, in __init__
    super(GFile, self).__init__(name, mode, _Pythonlocker())
  File "FILE_PATH_IN_MT_PC/tensorflow/python/platform/gfile.py", line 63, in __init__
    self._fp = open(name, mode)
TypeError: coercing to Unicode: need string or buffer, file found

I would be so happy if someone helps me to find the solution for this. Thanks in advance

Upvotes: 1

Views: 260

Answers (1)

LKS
LKS

Reputation: 673

I think in the open, your provide a file type object instead of a string for the name variable.

I do more digging:

In images = numpy.array(mnist.extract_images(f)), f is a file object.

But with tf.gfile.Open(filename, 'rb') as f, gzip.GzipFile(fileobj=f) as bytestream:, this treats the argument passed by images = numpy.array(mnist.extract_images(f)) as a filename.

This behavior does not appear in the latest version: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/learn/python/learn/datasets/mnist.py

Upvotes: 1

Related Questions