CODEWITHSUNDEEP
javamule
Stole
Stole

Reputation: 5318

How to limit number of files to be processed in a mule flow?

I have the following code:

   <sftp:connector name="ImportStatusUpdateSFTP" validateConnections="true" doc:name="SFTP"/>

   <flow name="UpdateFlow1" doc:name="UpdateFlow1">
    <sftp:inbound-endpoint sizeCheckWaitTime="${sftpconnector.sizeCheckWaitTime}" connector-ref="ImportStatusUpdateSFTP" host="${sftp.host}" port="${sftp.port}"
        path="${sftp.path}" user="${sftp.user}" password="${sftp.password}" responseTimeout="${sftp.responseTimeout}"
        archiveDir="${mule.servicefld}${sftp.archiveDir}" archiveTempReceivingDir="${sftpconnector.archiveTempReceivingDir}" archiveTempSendingDir="${sftpconnector.archiveTempSendingDir}"
        tempDir="${sftp.tempDir}" doc:name="SFTP" pollingFrequency="${sftp.poll.frequency}">
        <file:filename-wildcard-filter pattern="*.xml"/>
    </sftp:inbound-endpoint>    
            
    <idempotent-message-filter idExpression="#[headers:originalFilename]"
        throwOnUnaccepted="true" storePrefix="Idempotent_Message" doc:name="Idempotent Message"
        doc:description="Check for processing the same file again.">
        <simple-text-file-store name="FTP_files_names"
            maxEntries="1000" entryTTL="-1" expirationInterval="3600"
            directory="${mule.servicefld}${idempotent.fileDir}" />
    </idempotent-message-filter>

    <object-to-byte-array-transformer doc:name="Object to Byte Array"/>

    <message-filter onUnaccepted="Status_UpdateFlow_XML_Validation_Failed">
        <mulexml:schema-validation-filter schemaLocations="xsd/StatusUpdate.xsd" returnResult="false" doc:name="Schema_Validation"/>
    </message-filter>
    
    <vm:outbound-endpoint exchange-pattern="one-way"
        path="StatusUpdateIN" doc:name="StatusUpdateVMO" />
        
    <default-exception-strategy>
        <vm:outbound-endpoint path="serviceExceptionHandlingFlow" />
    </default-exception-strategy>           
  </flow>

My problem is, if there are lots of files on the SFTP (1000), it takes them all, converts them, validates them, and then sends them to the outbound-endpoint, and this puts a strain on the Application Processing Part.

Is there a way to limit, split, batch, filter or any other kind of action that will send only a max amount of messages / files to the outbound endpoint.

Upvotes: 0

Views: 393

Answers (1)

aled
aled

Reputation: 25872

In Mule 3 there is no built-in generic method to do this. There may be possible solutions in a case by case basis. In Mule 4 there is a simple way using the maxConcurrency attribute in flows.

Upvotes: 1

Related Questions