CODEWITHSUNDEEP
tensorflowgoogle-colaboratorytpugoogle-cloud-tpu
Rishabh Sahrawat
Rishabh Sahrawat

Reputation: 2507

TF1.14][TPU]Can not use custom TFrecord dataset on Colab using TPU

I have created a TFRecord dataset file consisting elements and their corresponding labels. I want to use it for training model on Colab using free TPU. I can load the TFRecord file and even run an iterator just to see the contents however, before the beginning of the epoch it throws following error-

UnimplementedError: From /job:worker/replica:0/task:0:
File system scheme '[local]' not implemented (file: '/content/gdrive/My Drive/data/encodeddata_inGZIP.tfrecord')
     [[{{node MultiDeviceIteratorGetNextFromShard}}]]
     [[RemoteCall]]
     [[IteratorGetNextAsOptional_1]]

In my understanding, it wants the TFRecord file on the TPU bucket, I don't know how to do that on Colab. How can one use a TFRecord file directly on Colab TPU?

Upvotes: 0

Views: 914

Answers (1)

Michielver
Michielver

Reputation: 116

You need to host it on Google Cloud Storage:

All input files and the model directory must use a cloud storage bucket path (gs://bucket-name/...), and this bucket must be accessible from the TPU server. Note that all data processing and model checkpointing is performed on the TPU server, not the local machine.

As mentioned on Google's troubleshooting page: https://cloud.google.com/tpu/docs/troubleshooting#cannot_use_local_filesystem

Hope this helps!

Upvotes: 1

Related Questions