CODEWITHSUNDEEP
google-cloud-storageairflowgoogle-cloud-composer
Bartłomiej Bęczkowski
Bartłomiej Bęczkowski

Reputation: 121

How to write to dags directory from Cloud Composer webserver?

I'm writing Cloud Composer plugin and I need to create DAG in runtime. How can I create DAG file from webserver or how can I access bucket ID from plugin code(so I can use gcs client and just upload DAG)? I tried code below and it doesn't work, I don't get any exceptions but also I don't see any results:

dag_path = os.path.join(settings.DAGS_FOLDER, dag_id + '.py')
with open(dag_path, 'w') as dag:
    dag.write(result)

Upvotes: 1

Views: 1143

Answers (2)

Neeraj
Neeraj

Reputation: 1817

You may either use the Environment Variables, or you may make use of the API's provided by GCloud SDK.

gcloud composer environments describe --format=json --project=<project-name> --location=<region> <cluster-name>

This would return the details of the cloud composer cluster. It would have the dag location under the key dagGcsPrefix

The format of dagGcsPrefix would be gs://<GCSBucket>/dags

Upvotes: 1

Bartłomiej Bęczkowski
Bartłomiej Bęczkowski

Reputation: 121

Possible solution is to read bucket ID from Cloud Composer env variable

Upvotes: 1

Related Questions