CODEWITHSUNDEEP
jsoncommand-line-interfaceairflow
PBartlett
PBartlett

Reputation: 103

Importing airflow variables in a json file using the command line

I have file called properties.json which looks like

{
  "FOO": {
    "var1": "apple",
    "settings": {
      "more_settings": {
        "var2": "pear",
        "var3": 123
      }
    }
  },
  "BAR": "FOO"
}

and I'm trying to import into Airflow via the command line interface like so

gcloud composer environments run <composer-name> \
        --location <location> --project <project> variables -- \
        --import /path/to/properties.json

in order to get the following variables into Airflow:

FOO: { "var1": "apple", "settings": { "more_settings": { "var2": "pear", "var3": 123 } } }

BAR: FOO

However, when I run the gcloud command, I get the error Missing variables file. When I import properties.json in Airflow's UI, it is imported without issues. What am I doing wrong by using the CLI?

Upvotes: 10

Views: 10834

Answers (3)

kevin_
kevin_

Reputation: 51

I have had this same issue with running Airflow on Docker. When running this command on Ubuntu:

sudo docker-compose run airflow-worker airflow variables import variable.json

I get this reponse:

Missing variables file.
ERROR: 1

I realized (finally) that I am getting this response because I'm running Airflow on Docker. My command above is giving a job to an Airflow worker to import a JSON file - but this worker has no access outside the container and so it doesn't 'see' the file I'm referring to at all.

The only solution I have now is simply to pass all values I need to use via the command itself. For example, this command works with no problems:

sudo docker-compose run airflow-worker airflow variables set my_var {\"test_key\":\"test_val\"}

However, I also just learned it is possible to set variables within the DAG file itself (more info here) and I might do that instead.

Upvotes: 0

Alex W
Alex W

Reputation: 38243

For older versions 1.10.x of Airflow, using CLI, the syntax is:

airflow variables -i <file.json>

Upvotes: 3

luisvenezian
luisvenezian

Reputation: 501

I have done this:

airflow variables import properties.json

According to: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#import_repeat2

Upvotes: 5

Related Questions