CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-functionsgoogle-kubernetes-engine
Aerocell
Aerocell

Reputation: 109

Triggering a training task on cloud ml when file arrives to cloud storage

I am trying to build an app where the user is able to upload a file to cloud storage. This would then trigger a model training process (and predicting later on). Initially I though I could do this with cloud functions/pubsub and cloudml, but it seems that cloud functions are not able to trigger gsutil commands which is needed for cloudml.

Is my only option to enable cloud-composer and attach GPUs to a kubernetes node and create a cloud function that triggers a dag to boot up a pod on the node with GPUs and mounting the bucket with the data? Seems a bit excessive but I can't think of another way currently.

Upvotes: 1

Views: 158

Answers (1)

aga
aga

Reputation: 3903

You're correct. As for now, there's no possibility to execute gsutil command from a Google Cloud Function:

Cloud Functions can be written in Node.js, Python, Go, and Java, and are executed in language-specific runtimes.

I really like your second approach with triggering the DAG. Another idea that comes to my mind is to interact with GCP Virtual Machines within Cloud Composer through the Python operator by using the Compute Engine Pyhton API. You can find more information in automating infrastructure and taking a deep technical dive into the core features of Cloud Composer here.

Another solution that you can think of is Kubeflow, which aims to make running ML workloads on Kubernetes. Kubeflow adds some resources to your cluster to assist with a variety of tasks, including training and serving models and running Jupyter Notebooks. Please, have a look on Codelabs tutorial.

I hope you find the above pieces of information useful.

Upvotes: 1

Related Questions