CODEWITHSUNDEEP
google-cloud-platformworkflowgoogle-cloud-composer
Kingvinst
Kingvinst

Reputation: 169

Google Cloud Composer: question regarding languages and workflows

I am about to write a sort of thesis about Google Cloud Composer and I faced some uncertainties.

In Google Cloud Composer the DAG is written in python. So does than mean I can only run python functions or can I also invoke e.g. java functions which I save in the google cloud?

If I stay with python, does everything has to be in one file? I guess this is a trivial question, but in all examples from GCC everything is just in one file. Also, it doesnt seem here that there is a folder for source code: https://cloud.google.com/composer/docs/concepts/cloud-storage

Finally, I am looking for compute intensive python workflow I can use. Can you recommend a site where I can find scientific python workflows?

Upvotes: 0

Views: 240

Answers (1)

Krish
Krish

Reputation: 812

The Cloud composer docs by Google uses Python only for writing DAGs.

Yes, there is a DAG folder which you can find in Composer environment. The folder lies in a bucket created by that Composer instance. If you want to execute multiple DAGs using same Composer instance, only thing you need to do is place the python files inside the DAGs folder in the bucket. Composer will automatically run that DAG. Adding a DAG

You can use various operators to add functionalities to your dag files like adding bashoperator to add java jar files LINK

everything has to be in one file?: Yes DAG file has all the workflows ie tasks defined in it. Sourcecode is stored in the DAG folder of the GCS bucket created by Airflow.

Upvotes: 1

Related Questions