CODEWITHSUNDEEP
transactionspentahojobskettlepentaho-data-integration
Kamil Gosciminski
Kamil Gosciminski

Reputation: 17137

Transactional job in Kettle Community Edition

Using: Pentaho Data Integration 5.4 Community Edition.

In documentation there is a statement that gives clarification about database transactions on a job level:

Transactions in jobs

Starting from version 5.0 of PDI (Enterprise Edition) you can also enable the same functionality that exists for a transformation in a job. The "Make the job database transactional" option is located in the "Settings" tab of the job settings dialog.

This is however only available for PDI Enterprise Edition. I'm in need for this feature in Community Edition.

I've been trying to do this inside transformations by using Transformation Executors, Mapping transformations with no success so far.

For a small process I've ended up using Blocking step inside a transformation and making the transformation database transactional.

Thinking bigger, as of ETL processes for data marts, data warehouses etc where in case of a failure full rollback of all processess is needed, how can one go around the issue of not having this feature in CE?

Edit: Some of you may have stumbled upon this issue and I'm wondering what would be a different way to approach this problem (excluding full database backup & restore - this will take forever with large databases like data warehouses).

Upvotes: 1

Views: 886

Answers (1)

Ken Clubok
Ken Clubok

Reputation: 1238

Have you tried out version 6.0? From the "What's New" page:

Revert changes in job database transactions

Database connections can be used with all jobs. This enables commits and rollbacks on a job level. Prior to this release, this was only possible with transformations.

I'm not sure, however, whether that functionality is available in CE.

Failing that, you could use the low-budget approach: At the beginning of the job, trigger a database backup. On a failure, restore the backup.

Upvotes: 1

Related Questions