CODEWITHSUNDEEP
python-3.xdockergoogle-cloud-platformgoogle-cloud-datastorecloud
Aida
Aida

Reputation: 2434

Google Cloud: Choosing the right storage option

I am developing a distributed application in Python. The application has two major packages, Package A and Package B that work separately but communicate with each other through a queue. In other words Package A generates some files and enqueue (pushes) them to a queue and Package B dequeues (pops) the files on a first-come-first-service basis and processes them. Both Package A and B are going to be deployed on Google Cloud as docker containers.

I need to plan what is the best storage option to keep the files and the queue. Files and the queue could be stored and used temporarily.

I think that my options are Cloud buckets or Google datastore, but have no idea how to choose from and what could be the best option. The best option would be a solution that has a low cost, reliable and easy-to-use from the development aspect.

Any suggestion is welcome... Thanks!

Upvotes: 0

Views: 65

Answers (1)

David
David

Reputation: 9721

Google Cloud Storage sounds like the right option for you because it supports large files. You have no need for features provided by datastore etc such as querying by other fields.

If you only need to process a file once, when it is first uploaded, you could use GCS pubsub notifications and trigger your processor from pubsub.

if you need more complex tasks, e.g. one task can dispatch to multiple child tasks that all operate on the same file, then it's probably better to use a separate task system like celery and pass the GCS URL in the task definition.

Upvotes: 2

Related Questions