CODEWITHSUNDEEP
airflowgoogle-sheets-apigoogle-cloud-composer
Lars Haugseth
Lars Haugseth

Reputation: 14881

Accessing sheets from operators on Google Cloud Composer 3

We are trying to migrate our DAGs from Airflow 2.9.3 on Composer 2 to Airflow 2.10.2 on Composer 3, but can't seem to get operators accessing Google Cloud Sheets to work.

What works on Composer 2, is to pass a custom connection to the operators using gcp_conn_id="google_cloud_sheets" with the following configuration:

{
    "conn_type": "google_cloud_platform",
    "description": "Connection for accessing Google Cloud Sheets",
    "extra": {
        "project": "{project_id}",
        "scope": "https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/spreadsheets.readonly",
        "num_retries": 5
    }
}

When trying to run these tasks on Composer 3, they fail with the following error in the logs:

<HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/{sheet_id}?alt=json returned "Request had insufficient authentication scopes."

We're using the exact same connection in both instances (stored in Secret Manager with the airflow-connections- prefix.).

I had a look at the source code for one of the failing operators (GoogleSheetsToGCSOperator) between Airflow versions, and there are no changes.

How can we get Composer 3 to pick up the additional scopes and pass them on to the Operator and the underlying API calls?

Upvotes: 0

Views: 66

Answers (0)

Related Questions