CODEWITHSUNDEEP
google-cloud-dataflow
CCC
CCC

Reputation: 2761

Keep executing dataflow pipeline after error

is there a way to keep executing a dataflow job even after some error happens? I'm reading from a CSV file to import into BigQuery, and one of the values in the CSV is a string while the column is of type integer, so I want the job to keep executing and ignore the problem. I actually got the error when trying to insert the first row, which is the header row, and then I thought that I wanted to just ignore all the bad rows and keep going.

Here is the error:

Could not convert value 'string_value: "XXXX"' to integer. Field: XXXX; Value: XXXX

I know one option is to prevent the error from occurring in the first place by trying to convert each value and in case of exception not try to include the row, but it seems to me that there should be a way to deal with BQ exceptions, which should be simpler.

Upvotes: 0

Views: 328

Answers (1)

jkff
jkff

Reputation: 17913

Currently BigQueryIO does not support this, however there's a JIRA you can follow to see the progress https://issues.apache.org/jira/browse/BEAM-1151.

Upvotes: 0

Related Questions