CODEWITHSUNDEEP
mysqlsql-server-2008talend
Aman
Aman

Reputation: 31

Talend job truncate records when i killed the job and run again

I am using the Talend open studio for Data Integration tool for transfer sql server table data to mysql server database.

I have a 40 million records into the table. I created and run the job but after inserting 20 million approx, connection failed.

When i Tried again to insert the Data then talend job firstly truncate the data from the table then it is inserting data from the beginning,

Upvotes: 0

Views: 590

Answers (2)

Vish
Vish

Reputation: 184

You can set the a property in context variable, say 'loadType' which will have value either 'initial' or 'incremental'. And before truncating table you should have 'if' link to check what is the value of this variable, if it is 'initial' it will truncate and it is 'incremental' then you can run your subjob to load data.

Upvotes: 0

prakul
prakul

Reputation: 39

The question seems to be incomplete, but assuming that you want the table not to truncate before each load, check the "Action on table" property. It should be set as "Default" or "Create Table if does not exist".

Now, if you're question is to handle restart-ability of the job where the job should resume from 20 million rows on the next run, there are multiple ways you could achieve this. In your case since you are dealing with high number of records, having a mechanism like pagination would help in which you load the data in chunks (lets say 10000 at a time) and loop it setting the commit interval as 10000. After each successful entry in the database of 10000 records, make an entry into one log table with the timestamp or incremental key in your data (to mark the checkpoint). Your job should look something like this: tLoop--{read checkpoint from table}--tMSSqlInput--tMySqlOutput--{load new checkpoint in table}

Upvotes: 1

Related Questions