CODEWITHSUNDEEP
google-cloud-platformairflowgoogle-cloud-composer
sag
sag

Reputation: 5461

Import variables using json file in Google Cloud Composer

How can I import a json file into Google Cloud Composer using command line?

I tried the below command

gcloud composer environments run comp-env --location=us-central1 variables -- --import composer_variables.json

I am getting the below error

[2019-01-17 13:34:54,003] {configuration.py:389} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-01-17 13:34:54,117] {app.py:44} WARNING - Using default Composer Environment Variables. Overrides have not been applied.
Missing variables file.

But when I set a single variable using below command it works fine.

gcloud composer environments run comp-env --location=us-central1 variables -- --set variable_name variable_value

Since I have more than 75 variables to be imported, we need to import it using json file. Please help me to resolve this issue

Upvotes: 6

Views: 6904

Answers (1)

Feng Lu
Feng Lu

Reputation: 691

The follow command gcloud composer environments run {environment-name} variables -- --i {path-to-json-file} executes airflow variables remotely inside the Airflow containes. Hence the json file needs to be accessible within the Airflow worker/scheduler pod. So you'll need to copy your var.json to GCS first and then run the command. For example:

  1. gcloud composer environments storage data import --source=your-var.json --environment={environment-name} --location={location}
  2. gcloud composer environments run {environment-name} --location={location} variables -- --i /home/airflow/gcs/data/your-var.json.

Upvotes: 13

Related Questions