CODEWITHSUNDEEP
talend
hopeIsTheonlyWeapon
hopeIsTheonlyWeapon

Reputation: 567

talend mysql bulk load can reject records and not fail?

I am using tmysqlbulk load to load data into mysql and just wanted to know can it silently reject records without fail ?

Currently loading data into a mysql table using mysql bulk load feature of talend. I am not seeing any error while the process is running and it completes with an exit code of 0. But can it reject records while processing them.

Upvotes: 0

Views: 1312

Answers (1)

E LaRoche
E LaRoche

Reputation: 1134

For bulk db operations Talend utilizes the native bulk load operations (of the db type you are using) called by JDBC. Talend does not incorporate all database options for each db type. Oracle is the most fully supported and allows you to ignore a configurable number of error on the bulk load. MySQL does not have even the option to ignore errors, never mind actually capturing the errors.

So there are three ways to work around this:

  1. The bulk output components provide the ability to utilize validation rules. If you are producing your own file for input into your MySQL database, then you can use validation rules in your OutputBulk component and ensure data quality prior to the input. When you utilize validation rules, a Validation Reject flow becomes available where you can capture the rejected records.

  2. the second option is to validate the file prior to importing it using the tSchemaComplianceCheck component. This too has a reject flow where you can capture rejected records. In this case you would use your bulk file as input, run it thru the validation to produce a 'good file' and a 'reject file'. You would then use your 'good file' as input to your bulk load. Of course, as a wrinkle on this option, you can avoid bulk load all together and perform not only the compliance check but all kinds of other validations, and use your input flows for the good records into a SQLOutput component.

    1. load your data into stage tables using bulk load. Then use ETL (either procs or Talend) to load your data into the destination tables, providing your data validation along the way.

Upvotes: 2

Related Questions