CODEWITHSUNDEEP
airflowgoogle-cloud-composer
JY2k
JY2k

Reputation: 2909

Cloud composer parse json

I have imported a simple json file into the data folder. What's the best way to load, parse and use params in the json?

Upvotes: 0

Views: 1099

Answers (1)

Aurélien Valade
Aurélien Valade

Reputation: 43

The gs://composer-bucket/data folder is fused to Airflow's /home/airflow/gcs/data directory (As specified here https://cloud.google.com/composer/docs/concepts/cloud-storage#folders_in_the_storage_name_bucket )

If you need to load it in some PythonOperator task definition for example, you could use the very simple following code:

import json
with open('/home/airflow/gcs/data/my_file.json', 'r') as f:
    d = json.load(f)

This is one way you can do it.

Depending on what you need to load and how you need to use it, you might want to look at Airflow Variables as well instead of a JSON file. (https://airflow.apache.org/concepts.html#variables)

Upvotes: 1

Related Questions