CODEWITHSUNDEEP
google-cloud-platformsshvirtual-machineairflowgoogle-compute-engine
Galuoises
Galuoises

Reputation: 3283

SSHOperator with ComputeEngineSSHHook

I am trying to run a command using ssh in a GCP VM in airflow via the SSHOperator as described here:

ssh_to_vm_task = SSHOperator(
    task_id="ssh_to_vm_task",
    ssh_hook=ComputeEngineSSHHook(
        instance_name=<MYINSTANCE>,
        project_id=<MYPROJECT>,
        zone=<MYZONE>,
        use_oslogin=False,
        use_iap_tunnel=True,
        use_internal_ip=False
        ),
    command="echo test_message",
    dag=dag
)

However, I get a airflow.exceptions.AirflowException: SSH operator error: [Errno 2] No such file or directory: 'gcloud' error.

Docker is installed via docker-compose following these instructions.

Other Airflow GCP operators (such as BigQueryCheckOperator) work correctly. So at first sight it does not seem like a configuration problem.

Could you please help me? Is this a bug?

Upvotes: 1

Views: 1297

Answers (2)

Galuoises
Galuoises

Reputation: 3283

It seems the issue is that gcloud was not installed in the docker container by default. This has been solved by following instructions in here: it is necessary to add

RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg  add - && apt-get update -y && apt-get install google-cloud-sdk -y

to the dockerfile that is used to install airflow / install dependencies.

Upvotes: 1

Leo
Leo

Reputation: 941

Check if the TCP port 22 is allowed through the firewall on your GCP VM instance, and make sure that the VM instance also allows SSH access and is properly configured in that VM instance. Furthermore, be sure that the IP from which you are trying to SSH at the VM instance is whitelisted through the firewall.

You can use the following command in GCP to check the ingress firewall rule for the network that contains the destination VM instance. Additionally, you can consult this [link]for more information.

This is an example of what you have to do. ´´´

gcloud compute firewall-rules list --filter network=[NETWORK-NAME] \
    --filter INGRESS \
    --sort-by priority \
    --format="table(
        name,
        network,
        direction,
        priority,
        sourceRanges.list():label=SRC_RANGES,
        destinationRanges.list():label=DEST_RANGES,
        allowed[].map().firewall_rule().list():label=ALLOW,
        denied[].map().firewall_rule().list():label=DENY,
        sourceTags.list():label=SRC_TAGS,
        sourceServiceAccounts.list():label=SRC_SVC_ACCT,
        targetTags.list():label=TARGET_TAGS,
        targetServiceAccounts.list():label=TARGET_SVC_ACCT
        )"

´´´

Upvotes: 0

Related Questions