CODEWITHSUNDEEP
google-cloud-platformdataform
Vandana D
Vandana D

Reputation: 66

How to run tasks which are after assertion in Google dataform even if assertion results in rows

I have few tasks to be run after an assertion check and my assertion results in more than zero rows due to which it's status is failed. My tasks after the assertion are getting skipped due to this reason. How can I continue with the tasks even if assertion is success or failure?

Upvotes: 2

Views: 213

Answers (1)

kyuden
kyuden

Reputation: 409

If you have no sqlx files that depend on the assertion, the failure will not stop the subsequent actions.

So, if your failing assertion is my_failing_assertion,

-- my_failing_assertion.sqlx
config {
  type: "assertion"
}

select * from some_table
where id is null

-- another_table.sqlx
config {
  type: "table",
  dependencies: ["my_failing_assertion"]
}

select * from my_source_table

This will fail, because another_table depends on my_failing_assertion, and the graph would look like this:

my_failing_assertion -> another_table

On the other hand, if you do not add the assertion as a dependency, the failure will not stop the workflow:

-- another_table.sqlx
config {
  type: "table"
}

select * from my_source_table

This is because the graph does not have an arrow (i.e. dependency) pointing from the assertion to the table.

my_failing_assertion
another_table

Note that the assertion itself can have a dependency without stopping the workflow:

-- my_failing_assertion.sqlx

config {
  type: "assertion",
  dependencies: ["some_table"]
}

select * from some_table
where id is null

(Although in the specific example above, you would probably better use ref("some_table") instead of declaring dependencies array; this is just for demonstration purposes)

Upvotes: 1

Related Questions