CODEWITHSUNDEEP
sqlake
Ajay C
Ajay C

Reputation: 66

In SQLake, if I stop my ingestion copy from job or my intermediate transformation job, what happens to my pipeline

Regular (non sync) Jobs are based on commit time. So if I stop any job, what happens to the dependent jobs. In current version, the entire pipeline would stop. Would Sqlake also behave same way or this would happen only for sync jobs?

Upvotes: 0

Views: 22

Answers (1)

jhall-upsolver
jhall-upsolver

Reputation: 11

With an "unsync'ed" job, if a COPY FROM job was stopped, all downstream transformation jobs would continue to run, but would not process any data. Once the COPY FROM job was restarted, and data ingested, all of the "newly" ingested data would have a $commit_time that would allow the downstream transformation jobs to process that data. In this case, $commit_time would be written as of the time the COPY FROM job was restarted, so the transformation jobs would include this data. This would also apply if you stopped intermediate transformation jobs.

Upvotes: 1

Related Questions