CODEWITHSUNDEEP
airflowgoogle-cloud-composer
SRK
SRK

Reputation: 1

BigQueryCheckOperator failed with 404 Error in Cloud composer

I am using BigQueryCheckOperator in Airflow to know if the data exists in BQ Table, but the Dag is failing with this ERROR - <HttpError 404 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/

Here is the logs of the Dag

Can Someone tell me how to fix this issue?

Upvotes: 0

Views: 1093

Answers (1)

Nick_Kh
Nick_Kh

Reputation: 5253

This is known Airflow issue querying Bigquery datasets residing in non multi-regional locations (US,EU) within some of Bigquery operator submodules, the pull request #8273 has been already raised.

You can also check out this Stack thread for most accurate problem explanation.

By now, it was announced to bring this problem fixed in Airflow 2.0 release, however community group has been pushed Backport package in order to help the users with older Airflow 1.10.* versions and it will be considered in further building Airflow images for GCP Composer.

Looking for a workaround, you can try to adjust BashOperator invoking bq command-line tool in attempt to perform the certain action against Bigquery dataset inside the particular DAG file.

Upvotes: 1

Related Questions