CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowapache-beam
crbl
crbl

Reputation: 397

Dataflow pipeline run error: SDK disconnect

I am trying to run a test Dataflow pipeline using DataflowRunner and --no_use_public_ips. It is reading a small table from Bigquery and write csv in storage, all in the same project.

python3 ./Dataflow/pipeline-v2.py   --project='<>'   --region='<>'   --service_account_email='<>'   --dataset_id='dataset_id'   --ingest_table_name='titanic'   --storage_bucket='<>'   --temp_location='<>'   --runner=DataflowRunner   --subnetwork='<>' --no_use_public_ips

I get this error:

INFO:apache_beam.runners.dataflow.dataflow_runner:2023-: JOB_MESSAGE_ERROR: Workflow failed. Causes: S40:Read from BigQuery/FilesToRemoveImpulse/Impulse+Read from BigQuery/FilesToRemoveImpulse/FlatMap()+Read from BigQuery/FilesToRemoveImpulse/Map(decode)+Read from BigQuery/MapFilesToRemove failed., The job failed because a work item has failed 4 times. Look in previous log entries for the cause of each one of the 4 failures. If the logs only contain generic timeout errors related to accessing external resources, such as MongoDB, verify that the worker service account has permission to access the resource's subnetwork. For more information, see https://cloud.google.com/dataflow/docs/guides/common-errors. The work item was attempted on these workers:

  Root cause: SDK disconnect.
  Worker ID: beamapp-root-...,

  Root cause: SDK disconnect.
  Worker ID: beamapp-root-...,

  Root cause: SDK disconnect.
  Worker ID: beamapp-root-...s,

  Root cause: SDK disconnect.
  Worker ID: beamapp-root-...

I suspect this problem is generated by some missing permissions from Dataflow to read from the Bigquery table but I am not sure if this is the cause. The service_account_email has all needed BQ permissions.

data = p | 'Read from BigQuery' >> beam.io.ReadFromBigQuery(query=query, use_standard_sql=True)

I do not know how to debug this. The only information I have is from the VM's CLI from where I initiate the pipeline and from the Cloud Logging.

enter image description here

Upvotes: 0

Views: 498

Answers (1)

crbl
crbl

Reputation: 397

Problem solved. One component was trying to import a library not present in the default worker image.

Upvotes: 0

Related Questions