CODEWITHSUNDEEP
google-bigquery
opensourcegeek
opensourcegeek

Reputation: 5982

Big query data storage cost

If the data is stored in Google's BigQuery, I can see how much I get charged on a monthly basis in calculator. Say if I save up 1 TB of data every month and if I leave the data for 2 years then will have 24 TB of stored data. Will that mean I'll be paying for 24TB worth of storage? I guess so - but the calculator doesn't make it very clear. Could some one please confirm this from Google's BigQuery team?

Upvotes: 1

Views: 536

Answers (1)

Patrice
Patrice

Reputation: 4692

It is off-topic, but in any case :

you pay for everything stored, so if you store 1 TB a month, you will end up paying 24TB in 2 years indeed.

You can (and maybe should) use an "expiration time" when you create your table, this way your tables clear automatically when enough time has elapsed.

You could also have a cron job on App Engine that runs over your TB of data and "compresses it" (I guess after a month, you don't need every single row of data, but rather statistics on it, which you could create before deleting the table with the data).

Also consider doing backups of your data, compressing them, then storing them on Google Cloud Storage, and restoring them as needed (indeed, you pay for TB/months. So if you load 24 TBs for one day, you don't even pay a full TB month). Do note there are fees with storing in GCS though, so you'd have to study the full usage to make sure this will be worth it.

Upvotes: 3

Related Questions