CODEWITHSUNDEEP
google-cloud-dataflowgoogle-cloud-build
user1068378
user1068378

Reputation: 319

Cloud Builds Failulre , unable to find logs to see what is going on

i am kicking off a dataflow flex template using a cloud build. In my cloud build file i am attempting to do 3 things

substitutions:
  _IMAGE: my_logic:latest4
  _JOB_NAME: 'pipelinerunner'
  _TEMP_LOCATION: ''
  _REGION: us-central1
  _FMPKEY: ''  
  _PYTHON_VERSION: '3.8'

# checkout this  link https://github.com/davidcavazos/python-docs-samples/blob/master/dataflow/gpu-workers/cloudbuild.yaml

steps:
- name: gcr.io/cloud-builders/docker
  args:
    [ 'build'
    , '--build-arg=python_version=$_PYTHON_VERSION'
    , '--tag=gcr.io/$PROJECT_ID/$_IMAGE'
    , '.'
    ]

# Push the image to Container Registry.
- name: gcr.io/cloud-builders/docker2
  args: [ 'push', 'gcr.io/$PROJECT_ID/$_IMAGE' ]



- name: gcr.io/$PROJECT_ID/$_IMAGE
  entrypoint: python
  args:
  - /dataflow/template/main.py
  - --runner=DataflowRunner
  - --project=$PROJECT_ID
  - --region=$_REGION
  - --job_name=$_JOB_NAME
  - --temp_location=$_TEMP_LOCATION
  - --sdk_container_image=gcr.io/$PROJECT_ID/$_IMAGE
  - --disk_size_gb=50
  - --year=2018 
  - --quarter=QTR1 
  - --fmpkey=$_FMPKEY
  - --setup_file=/dataflow/template/setup.py  

options:
  logging: CLOUD_LOGGING_ONLY

# Use the Compute Engine default service account to launch the job.
serviceAccount: projects/$PROJECT_ID/serviceAccounts/$PROJECT_NUMBER-compute@developer.gserviceaccount.com

And this is the command i am launching

gcloud beta builds submit \
    --config run.yaml \
    --substitutions _REGION=$REGION \
    --substitutions _FMPKEY=$FMPKEY \
    --no-source

The error message i am getting is this

Logs are available at [https://console.cloud.google.com/cloud-build/builds/0f5953cc-7802-4e53-b7c4-7e79c6f0d0c7?project=111111111].
ERROR: (gcloud.beta.builds.submit) build 0f5953cc-7802-4e53-b7c4-7e79c6f0d0c7 completed with status "FAILURE

but i cannot access the logs from the URL mentioned above

I cannot see the logs, so i am unable to see what is wrong, but i stongly suspect somethign in my run.yaml is not quite right Note: before this, i was building the image myself by launching this command

gcloud builds submit --project=$PROJECT_ID --tag $TEMPLATE_IMAGE  .

and my run.yaml just contained 1 step, the last one, and everything worked fine But i am trying to see if i can do everything in the yaml file Could anyone advise on what might be incorrect? I dont have much experience with yaml files for cloud build thanks and regards Marco

Upvotes: 0

Views: 1149

Answers (2)

user1068378
user1068378

Reputation: 319

Ok, sorted, the problem was the way i was launching the build command

this is the original

gcloud beta builds submit \
    --config run.yaml \
    --substitutions _REGION=$REGION \
    --substitutions _FMPKEY=$FMPKEY \
    --no-source

apparently when i removed the --no-source all worked fine. I think i copied and pasted the command without really understanding it

regards

Upvotes: 1

Iñigo González
Iñigo González

Reputation: 3955

I guess the pipeline does not work because (in the second step) the container: gcr.io/cloud-builders/docker2 does not exist (check https://gcr.io/cloud-builders/ - there is a docker container, but not a docker2one).

This second step pushes the final container to the registry and, it is a dependence of the third step, which will fail too.

You can build the container and push it to the container registry in just one step:

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/$IMAGE_NAME', '<path_to_docker-file>']
images: ['gcr.io/$PROJECT_ID/$IMAGE_NAME']

Upvotes: 1

Related Questions