CODEWITHSUNDEEP
loopswso2mediatorsmookswso2-esb
RudyH
RudyH

Reputation: 1

WSO2 ESB processing a large file, stopping and restarting?

Is it possible in the WSO2 ESB to process a large file (with each line representing a single record/message), and stop or pause the processing within the file, then restart where it left off? (using the Smooks mediator and/or Iterate mediator, so any other mechanism)

It appears that if you are processing a large file (say with 10K entries) you cannot stop or pause the processing (or say the ESB is brought down in the middle of processing the file), and then restart where it was stopped. Upon restarting either the whole file has to be reprocessed, or the file is dumped to the error folder and skipped. Is this correct?

Thanks for any help on this.

Upvotes: 0

Views: 562

Answers (1)

chamilad
chamilad

Reputation: 1669

AFAIK this cannot be done in a straight forward manner. However you could probably split the large file in to smaller files using Smooks mediator (message splitting) [1], and use VFS processing on it [2]. Each file on the VFS location that is processed will either be deleted or moved, so whenever the process restarts it will not process the already processed messages/files.

[1] - https://github.com/smooks/smooks/tree/v1.5.1/smooks-examples/file-router/

[2] - https://docs.wso2.com/pages/viewpage.action?pageId=33136056

Upvotes: 1

Related Questions