GCP Vertex AI Training Custom Job : User does not have bigquery.jobs.create permission

Question

I'm struggling to execute a query with Bigquery python client from inside a training custom job of Vertex AI from Google Cloud Platform.

I have built a Docker image which contains this python code then I have pushed it to Container Registry (eu.gcr.io)

I am using this command to deploy

      gcloud beta ai custom-jobs create --region=europe-west1 --display-name="$job_name" \
        --config=config_custom_container.yaml \
        --worker-pool-spec=machine-type=n1-standard-4,replica-count=1,container-image-uri="$docker_img_path" \
        --args="${model_type},${env},${now}"  

I have even tried to use the option --service-account to specify a service account with admin Bigquery role, it did not work.

According to this link https://cloud.google.com/vertex-ai/docs/general/access-control?hl=th#granting_service_agents_access_to_other_resources

the Google-managed service accounts for AI Platform Custom Code Service Agent (Vertex AI) have already the right to access to BigQuery, so I do not understand why my job fails with this error

google.api_core.exceptions.Forbidden: 403 POST https://bigquery.googleapis.com/bigquery/v2/projects/*******/jobs?prettyPrint=false: 
Access Denied: Project *******:
User does not have bigquery.jobs.create permission in project *******.

I have replaced the id with *******

---

Edit: I have tried several configuration, my last config YAML file only contents this

baseOutputDirectory:
  outputUriPrefix:

Using the field serviceAccount does not seem to edit the actual configuration unlike --service-account option

---

Edit 14-06-2021 : Quick Fix

like @Ricco.D said

try explicitly defining the project_id in your bigquery code if you have not done this yet.

bigquery.Client(project=[your-project])

has fixed my problem. I still do not know about the causes.

Answer 1

To fix the issue it is needed to explicitly specify the project ID in the Bigquery code.

Example:

bigquery.Client(project=[your-project], credentials=credentials)