CODEWITHSUNDEEP
google-colaboratorygoogle-cloud-tpu
David Nemeskey
David Nemeskey

Reputation: 640

Colab tells me to create a bucket, but where?

When using TPUs on Google Colab (such as in the MNIST example), we are told to create a GCS bucket. However, it doesn't tell us where. Without knowing the region/zone of the Colab instance, I am afraid to create a bucket in fear of running into billing issues.

There are actually several questions:

  1. Is accessing a GCS bucket from Colab free, or do the normal network egress fees apply?
  2. Can I get the region/zone of the colab instance? Most likely not.
  3. If the question to both questions above is "no": is there any solution for minimizing costs when using TPUs with Colab?

Upvotes: 4

Views: 749

Answers (2)

Rub
Rub

Reputation: 2708

We are told to create a GCS bucket. However, it doesn't tell us where.

Running (within Colab)

!curl ipinfo.io

You get something similar to

{
  "ip": "3X.20X.4X.1XX",
  "hostname": "13X.4X.20X.3X.bc.googleusercontent.com",
  "city": "Groningen",
  "region": "Groningen",
  "country": "NL",
  "loc": "53.21XX,6.56XX",
  "org": "AS396XXX Google LLC",
  "postal": "9711",
  "timezone": "Europe/Amsterdam",
  "readme": "https://ipinfo.io/missingauth"
}

Which tells you where you Colab is running.

You can create a GCS bucket in just one region (if you don't need multi-region).

Assuming you don't change country/area very often, you can check that a few times (different days) and get an idea of where your Colab is probably going to be located.

For your other questions (egress,...) see the Conclusion on https://ostrokach.gitlab.io/post/google-colab-storage/

[...] Google Cloud Storage is a good option for hosting our data. Only we should be sure to check that the Colab notebook is running in the same continent as your Cloud Storage bucket, or we will incur network egress charges!

Upvotes: 0

aman2930
aman2930

Reputation: 275

Thank you for your question.

No, you can not get the region/zone of the colab instance. So you can try creating a multi-regional GCS bucket which should be accessible by the colab. As per the comment, https://github.com/googlecolab/colabtools/issues/597#issuecomment-502746530, Colab TPU instances are only in US zone. So while creating a GCS bucket, you can choose a Multi-region bucket in US.

Checkout https://cloud.google.com/storage/pricing to get more details about the pricing information for the GCS buckets.

You can also sign up for a Google Cloud Platform account with 5GB of free storage and $300 in credits at https://cloud.google.com/free/, so that should be able to provide you with enough credits to get started.

Upvotes: 5

Related Questions