CODEWITHSUNDEEP
pythontensorflowgoogle-cloud-platformgoogle-cloud-storagegoogle-cloud-datalab
LogicChains
LogicChains

Reputation: 4412

Is it possible to create a TFRecordDataset from Google Cloud Storage Objects?

I'm testing a Tensorflow model in a Jupyter notebook on Google Cloud Datalab. The model is set up to use a TFRecordDataset of TFRecord files as input, which I'm led to believe is the recommended way of organising input to a model. I was wondering if there's some supported way of creating this TFRecordDataset from a list of Google Cloud Storage Objects, which are provided by the google.datalab.storage API as a means of accessing data on Google Cloud storage buckets.

Edit: I found it seems possible by calling .uri on the Storage Objects and passing those uris to the TFRecordDataset, but I'm not sure if this is the recommended way.

Upvotes: 2

Views: 681

Answers (2)

visionEnthusiast
visionEnthusiast

Reputation: 67

Typically such task would have settings where the Google Cloud Storage would have data (let's say images) stored in the buckets and user would want to convert the inputs into tfrecord format in order to train/finetune a model (let's say classification/detection). For such a task you will have to create a file containing information regarding image paths and suitable other information such as labels, bbox etc. Once you have created that file, you can iterate over your file, and using the google storage paths of those input files access the GCS data to process it and save it as a tfrecord. It is useful to create a file in a popular formats (COCO/Pascal VOC for image data), since you can utilize premade conversion scripts which are less error-prone and faster. For example: COCO to TFrecord conversion

Upvotes: 0

Salma R
Salma R

Reputation: 204

Your question is a little unclear, but here is how I converted my input data to tfrecord in order to feed it into my model.

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/how_tos/reading_data/convert_to_records.py

You can read your data from google storage in Datalab, and convert them to tfrecord, and save them back to google storage.

Upvotes: -1

Related Questions