CODEWITHSUNDEEP
airflow
Mohammed
Mohammed

Reputation: 43

xcom_pull from the GoogleCloudStoragePrefixSensor

I want the file name from the GoogleCloudStoragePrefixSensor Operator. Tried using context['ti'].xcom_pull(task_ids='check_file_gcs') where the main check_file_gcs is

check_file_gcs = GoogleCloudStoragePrefixSensor(
        task_id='check_file_gcs',  
        bucket=source_bucket,
        prefix="<file_name>",
        poke='10',
        dag=dag)

I am not able to get the file name from the xcom_pull. Not much experienced in Airflow. Can someone help please? How do we know the paramneters and values passed by the completion of GoogleCloudStoragePrefixSensor.

Thanks

Upvotes: 1

Views: 1628

Answers (1)

Elad Kalif
Elad Kalif

Reputation: 15979

GoogleCloudStoragePrefixSensor is deprecated and doesn't have the functionality of pushing matching objects to xcom. You should use the new sensor GCSObjectsWithPrefixExistenceSensor.

Since you are using the deprecated version it means you are using Airflow < 2.0.0 thus you will need to install google backport provider:

pip install apache-airflow-backport-providers-google

Then you can import the sensor with:

from airflow.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensor

To verify that the matching objects are pushed to xcom you can click on the GCSObjectsWithPrefixExistenceSensor in the UI and choose Xcom tab. The list of objects will appear there.

Upvotes: 1

Related Questions