CODEWITHSUNDEEP
wso2etlsmooks
adnecs
adnecs

Reputation: 175

Simple ETL: Smooks or ETL product

I am fairly new to the subject and doing some research. I have an ESB (using WSO2 ESB) and want to extract master data from the passing messages (like Customers, Orders, etc) and store them in DB to keep as a reference data. Source data is in XML coming from web services. So there needs to be a component that will be able to maintain master data: insert new objects, delete old and update changed (would be also nice to have data events so ESB can route data accordingly).Basically, the logic will be similar for any entity type and it might be good idea to autogenerate it for all new entity types...

Options as I see them now:

  1. Use Smooks with either SQLExecutor or Hibernate for persistence with all matching logic written either in smooks config or in DAO annotations
  2. Use some open source ETL tool (like Talend, Kettle, Clover, etc). So the data will be passed to the ETL and all transformation logic is defined there. Also could accommodate future scenarios when they appear or can be an overkill..

.

Would appreciate if you share your thoughts and point me to the right direction.

Upvotes: 1

Views: 1206

Answers (1)

centr
centr

Reputation: 655

You'd better to leave your database part to another tool.

If you have a fair amount of database interactions in your message flow, you can expect serious decreases in your performance.

However you do not need an ETL for the use case you explained. You can simply do it using WSO2 DSS by creating services to insert or update your data inside the database.

We have been using this for message logging purposes (inside DB) beside the ESB and are happy with that. It's better to use it as non-blocking fire-and-forget web services in your message flow within ESB. Hope this helps.

Upvotes: 0

Related Questions