CODEWITHSUNDEEP
pysparkdatabricksdelta-lakedelta-live-tables
Leonardo Lima
Leonardo Lima

Reputation: 93

How to obtain a direct way to differentiate between a full refresh and an incremental update for Delta live table?

I have a tables that travels from Bronze - silver - gold,

I want to implement some function like 'is_full_refresh()' so the pipeline filters the df depending on the output, if it's a full, don't filter, if it's incremental filter by a,b,c

Checking the documentation on Databricks https://docs.databricks.com/delta-live-tables/settings.html#cluster-config can't find a direct way to differentiate between a full refresh and an incremental,

How can I do that?

Upvotes: 1

Views: 720

Answers (1)

partlov
partlov

Reputation: 14277

One option would be to rely on REST API call https://docs.databricks.com/api/workspace/pipelines/getupdate.

Another option I would try (requires some investigation) is to query DLT event logs: https://docs.databricks.com/delta-live-tables/observability.html. I guess some log events may have this info.

Upvotes: 1

Related Questions