CODEWITHSUNDEEP
javaspringspring-integration
robiulh
robiulh

Reputation: 23

Issues with Spring Integration and process taking time and pausing

I am looking at some issue that we have in our application. Spring integration is being used to poll a particular directory and then process the files in this directory. It can process 5k 1kb files and sometimes there is a huge pause where the application is doing nothing just sitting idle and then completes the process in 4 minutes. Then the next run will take a bit longer and the one after that takes slightly longer and so on until i restart the application where it goes back to the 4 minutes mark. Has anyone experienced this issue before.

I wrote a standalone version without Spring Integration and dont get the same issue. I have also below pasted the xml config, just incase i have done something wrong that I can't spot.

Thanks in advance.

 <!-- Poll the input file directory for new files.  If found, send a Java File object on inputFileChannel -->
    <file:inbound-channel-adapter directory="file:${filepath}"
        channel="inputFileChannel" filename-regex=".+-OK.xml">
        <si:poller fixed-rate="5000" max-messages-per-poll="1" />
    </file:inbound-channel-adapter>

    <si:channel id="inputFileChannel" />

    <!-- Call processFile() and start parsing the XML inside the File -->
    <si:service-activator input-channel="inputFileChannel"
                          method="splitFile" ref="splitFileService">
    </si:service-activator>

    <!-- Poll the input file directory for new files.  If found, send a Java File object on inputFileChannel -->
    <file:inbound-channel-adapter directory="file:${direcotrypath}" channel="inputFileRecordChannel" filename-regex=".+-OK.xml">

 <si:poller fixed-rate="5000" max-messages-per-poll="250" task-executor="executor" />
    </file:inbound-channel-adapter>

    <task:executor id="executor" pool-size="8"
                   queue-capacity="0"
                   rejection-policy="DISCARD"/>

    <si:channel id="inputFileRecordChannel" />

    <!-- Call processFile() and start parsing the XML inside the File -->
    <si:service-activator input-channel="inputFileRecordChannel"
                          method="processFile" ref="processedFileService">
    </si:service-activator>

    <si:channel id="wsRequestsChannel"/>

    <!-- Sends messages from wsRequestsChannel to the httpSender, and returns the responses on
         wsResponsesChannel.  This is used once for each record found in the input file. -->
    <int-ws:outbound-gateway uri="#{'http://localhost:'+interfaceService.getWebServiceInternalInterface().getIpPort()+'/ws'}"
        message-sender="httpSender"
        request-channel="wsRequestsChannel" reply-channel="wsResponsesChannel" mapped-request-headers="soap-header"/>

    <!-- Handles the responses from the web service (wsResponsesChannel).  Again
         this is used once for each response from the web service -->
    <si:service-activator input-channel="wsResponsesChannel"
        method="handleResponse" ref="responseProcessedFileService">
    </si:service-activator>

Upvotes: 2

Views: 1010

Answers (1)

Gary Russell
Gary Russell

Reputation: 174809

As I surmised in the comment to your question, the (default) AcceptOnceFileListFilter does not scale well for a large number of files because it performs a linear search over the previously processed files.

We can make some improvements there; I opened a JIRA Issue for that.

However, if you don't need the semantics of that filter (i.e. your flow removes the input file on completion), you can replace it with another filter, such as an AcceptAllFileListFilter.

If you need accept once semantics you will need a more efficient implementation for such a large number of files. But I would warn that when using such a large number of files, if you don't remove them after processing, things are going to slow down anyway, regardless of the filter.

Upvotes: 0

Related Questions