CODEWITHSUNDEEP
google-cloud-platformairflowslack-apigoogle-cloud-composer
sacoder
sacoder

Reputation: 189

Slack Integration with Cloud Composer - broken DAG No module named 'slackclient'

We are having trouble integrating slack with Cloud composer. The SlackAPIPostOperator has slackclient dependency, which we installed directly on the worker nodes using below:

sudo python3 -m pip install apache-airflow[slack]

This installs slack 1.3.2.

We had to install on the worker nodes since the pypi package install option on the Cloud composer UI was repeatedly timing out.

The module import statements work fine on the worker nodes. However, composer still reports broken DAGs with the error message: Broken DAG: [/home/airflow/gcs/dags/composer-slack-test.py] No module named 'slackclient'

The Composer version is: composer-1.16.0-airflow-1.10.15

Any help is appreciated.

Thanks!

Upvotes: 0

Views: 1232

Answers (1)

hexacyanide
hexacyanide

Reputation: 91769

If the package installation times out, it likely means there was a package conflict when Cloud Composer attempted to build the worker/scheduler images. To troubleshoot this, you should look at the build logs after the update operation times out, and look for the output of pip check. There, you will be able to identify which packages were unable install due to version constraints.

Packages installed on worker nodes will be not be accessible from workflows because the Airflow worker/scheduler run as Kubernetes pods. You should not manually install packages onto nodes or worker/scheduler pods, because they can be replaced at any time.

Upvotes: 1

Related Questions