CODEWITHSUNDEEP
dockerdocker-composeairflowairflow-scheduler
Shadow Walker
Shadow Walker

Reputation: 1199

How to access Airflow 2.0 cli

I'm running airflow using docker and I want to use the cli to perform pause and unpause a dag since I'm unable to install a browser on a remote Digital Ocean droplet server in order to access airflow UI.

At the moment I'm trying to access the airflow cli with the command sudo docker exec -it $(docker-compose -f docker-compose.yaml ps -q webserver) bash but I'm getting an error

docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied')) [95826] Failed to execute script docker-compose "docker exec" requires at least 2 arguments.

below is my docker-compose.yaml file

version: '3'
x-airflow-common:
  &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.1}
  environment:
    &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CORE__FERNET_KEY: ''
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
    AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
  volumes:
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./plugins:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
  depends_on:
    postgres:
      condition: service_healthy

services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
      POSTGRES_DB: airflow
    volumes:
      - postgres-db-volume:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "airflow"]
      interval: 5s
      retries: 5
    restart: always

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 8081:8080
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 10s
      timeout: 10s
      retries: 5
    restart: always

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    restart: always

  airflow-init:
    <<: *airflow-common
    command: version
    environment:
      <<: *airflow-common-env
      _AIRFLOW_DB_UPGRADE: 'true'
      _AIRFLOW_WWW_USER_CREATE: 'true'
      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}

volumes:
  postgres-db-volume:

How do i go about fixing this?

Upvotes: 5

Views: 4056

Answers (2)

Jarek Potiuk
Jarek Potiuk

Reputation: 20067

If you look at our recent quick-start docker compose, you will see that you can add a separate `airflow-cli' docker compose (it will be released tomorrow / day after with 2.1.4 but you can see it here:

https://github.com/apache/airflow/blob/v2-1-stable/docs/apache-airflow/start/docker-compose.yaml#L229

  airflow-cli:
    <<: *airflow-common
    profiles:
      - debug
    environment:
      <<: *airflow-common-env
      CONNECTION_CHECK_MAX_COUNT: "0"
    # Workaround for entrypoint issue. See: https://github.com/apache/airflow/issues/16252
    command:
      - bash
      - -c
      - airflow

If you have this added to your docker-compose, you can run docker-compose run airflow-cli <command> (and other commands)

The debug profile makes sure that the airflow-cli service is not started by default, only when you specify --profile debug or specify the service to run explicitely.

Upvotes: 11

Bas Harenslak
Bas Harenslak

Reputation: 3064

The quick fix is probably to run sudo docker exec -it $(sudo docker-compose -f docker-compose.yaml ps -q webserver) bash (notice the sudo for docker-compose).

That said, it's good practice to avoid sudo if possible, here's how to run docker commands without: https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user.

Upvotes: 0

Related Questions