Binary file processing using ESBs (WSO2, Mule, Petals, Service Mix)

Question

I'm keen to use an ESB (probably WSO2, Mule, Petals or Service Mix) on a new project.

I have a number of business processes/ workflows which will work well in an ESB: however I'm struggling with one particular scenario.

The basic requirement is to process incoming binary files through a number of stages:

• pre-processing the file using an external tool;
• splitting the file based on some configuration data;
• processing each of these smaller files to calculate a "score". Some of this will be handled by code I will write, but external software will also be called in some instances;
• loading all the calculated scores into a database. This must be done at the end of the flow rather than incrementally (as the primary key will be only be found during processing of the file);
• moving the initial file to an archive area.

The incoming files will be dropped into a local directory. All the tools support polling of a directory so this isn't a problem.

However, from what I've read most of the tools expect the payload of the message to be easily readable so that processing can be performed directly on the payload. In my case I merely want to use the incoming file to generate my real payload data: the values which I will write to the database.

Is this a pattern that ESBs can support? Are there any articles about this kind of data processing with any of the above tools? Should I be looking at any other technology/patterns?

Hope this makes sense!

Many thanks,

Ken

Answer 1

This should be possible with an ESB. In particular with UltraESB you could do this with its built-in file transport, which supports the moving of the file after processing to an archive area by default.

The Splitting and aggregation features of the UltraESB could be used to split the file into parts and process them separately. The "score" generation can be done with Java or any other scripting language which leverage the UltraESB API to interact with the message (in this case the part of the file) and its environment.

The DB interactions can be easily done with the UltraESB with its data base connection mechanisms. Further, you could do these updates transactionally with UltraESB.

If the file is of a standard CSV or EDI format, then you can use the csv or EDI support to process the file easily. All these via an extremely easy configuration with either Java or any scripting language.

[1] - http://docs.adroitlogic.org/display/esb/Transactional+ESB+use+cases+made+simple+with+the+UltraESB

[2] - http://docs.adroitlogic.org

Answer 2

WSO2 provides capability to listen to the files in the file system. However, as per your query it seems you need to process a binary file with your proprietary format. If so, you may use WSO2 ESB vfs transport[1] and you have to implement a custom message formatter and a builder (trivial java code as described on [2])

[1] http://wso2.org/project/esb/java/4.0.0/docs/samples/transport_samples.html#Sample254 [2] http://charithwiki.blogspot.com/2010/11/how-to-write-axis2-message-builder.html