CODEWITHSUNDEEP
google-cloud-composer
Josh Laird
Josh Laird

Reputation: 7234

Cloud Composer missing variables file

I've been trying to import a JSON file of environment variables to a newly created Cloud Composer instance using the airflow CLI but when running the below I get the error: Missing variables file.

gcloud composer environments run ${COMPOSER_NAME} \
    --location=${COMPOSER_LOCATION} \
    variables -- \
    -i ${VARIABLES_JSON}

From looking at the source it seems that this happens when an environment variable file doesn't exist at the expected location. Is this because Cloud Composer sets up its variables in a different location so this CLI won't work? I've noticed that there's a env_var.json file that's created on the instance's GCS bucket, I realise I can overwrite this file but that doesn't seem like best practice.

Upvotes: 3

Views: 1464

Answers (1)

Josh Laird
Josh Laird

Reputation: 7234

It feels like a hack but I copied over the variables.json to my Composer's GCS bucket data folder and then it worked.

This is due to os.path.exists() checking the container that Airflow is running on. I chose this approach over overwriting env_var.json because I get the variables in Airflow's UI with this method.

Script for anyone interested:

COMPOSER_DATA_FOLDER=/home/airflow/gcs/data

COMPOSER_GCS_BUCKET=$(gcloud composer environments describe ${COMPOSER_NAME} --location ${COMPOSER_LOCATION} | grep 'dagGcsPrefix' | grep -Eo "\S+/")

gsutil cp ${ENV_VARIABLES_JSON_FILE} ${COMPOSER_GCS_BUCKET}data

gcloud composer environments run ${COMPOSER_NAME} \
    --location ${COMPOSER_LOCATION} variables -- \
    -i ${COMPOSER_DATA_FOLDER}/variables.json

Upvotes: 3

Related Questions