Mule Datamapper Streaming hangs forever

Question

I am working with Mule EE 3.5.2, trying to stream payload through:

1. Http request-response endpoint that has content-type in application/xml and output in type ReleasingInputStream

2. flow data into a Datamapper (stream switched on), map from XML to Collection

3. then use For Each scope to process each pojo element, using jaxb processcor to transform them into individual xml

4. send each xml into a jms queue

When I'm testing with small portion of data, this worked all good, but once I hook the real datasource in, with roughly 600,000 records flowing out of the http endpoint, the data mapper hangs with below in log, and no more action happens, please help me to find out why, thanks.

My Flow xml (abstract):

<flow>
<http:outbound-endpoint exchange-pattern="request-response"
                    method="GET" 
                    doc:name="GET /v1/student/detail"
                    password="${esb.service.password.mule}" user="${esb.service.user.mule}"
                    host="${service-v1.host}" mimeType="application/xml"
                    path="v1/resource/detail?lastmodifiedsince=#[java.net.URLEncoder.encode(flowVars['studentone-studentdetails-watermark'], 'UTF-8')]"
                    port="${service-v1.port}"
                    responseTimeout="${service-v1.timeout}" />

        <data-mapper:transform doc:name="Split StudentOne StudentItems" config-ref="mapping_ref" stream="true" pipeSize="2048"/>

<foreach doc:name="For Each">
            <mulexml:jaxb-object-to-xml-transformer returnClass="java.lang.String" jaxbContext-ref="jaxbContext" doc:name="JAXB Object to XML"/>

            <jms:outbound-endpoint"/>
        </foreach>

</flow>

Log output, that hangs at last line:

 DEBUG 2015-07-03 00:11:25,122 [WatchDog_0] org.jetel.graph.runtime.WatchDog: XML_READER0 ... starting
DEBUG 2015-07-03 00:11:25,127 [WatchDog_0] org.jetel.graph.runtime.WatchDog: All components are ready to start.
INFO  2015-07-03 00:11:25,127 [WatchDog_0] org.jetel.graph.runtime.WatchDog: Successfully started all nodes in phase!
INFO  2015-07-03 00:11:25,319 [main] org.jetel.graph.runtime.EngineInitializer: Checking graph configuration...
INFO  2015-07-03 00:11:25,356 [main] org.jetel.graph.runtime.EngineInitializer: Graph configuration is valid.
INFO  2015-07-03 00:11:25,357 [main] org.jetel.graph.runtime.EngineInitializer: Graph initialization (Split_StudentOne_StudentItems)
DEBUG 2015-07-03 00:11:25,357 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge3] : BUFFERED
DEBUG 2015-07-03 00:11:25,357 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge9] : BUFFERED
DEBUG 2015-07-03 00:11:25,357 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge7] : BUFFERED
DEBUG 2015-07-03 00:11:25,358 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge1] : BUFFERED
DEBUG 2015-07-03 00:11:25,358 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge5] : BUFFERED
DEBUG 2015-07-03 00:11:25,358 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge11] : BUFFERED
DEBUG 2015-07-03 00:11:25,358 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge0] : BUFFERED
DEBUG 2015-07-03 00:11:25,358 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge2] : BUFFERED
DEBUG 2015-07-03 00:11:25,358 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge4] : BUFFERED
DEBUG 2015-07-03 00:11:25,358 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge6] : BUFFERED
DEBUG 2015-07-03 00:11:25,358 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge10] : BUFFERED
DEBUG 2015-07-03 00:11:25,358 [main] org.jetel.graph.TransformationGraph: EdgeType [Edge8] : BUFFERED
INFO  2015-07-03 00:11:25,359 [main] org.jetel.graph.Phase: Initializing phase 0
DEBUG 2015-07-03 00:11:25,360 [main] org.jetel.graph.Phase: Initializing edges
DEBUG 2015-07-03 00:11:25,385 [main] org.jetel.graph.Phase: All edges initialized successfully.
DEBUG 2015-07-03 00:11:25,385 [main] org.jetel.graph.Phase: Initializing nodes
DEBUG 2015-07-03 00:11:25,421 [main] org.jetel.graph.Phase:     BEAN_WRITER0 ...OK
DEBUG 2015-07-03 00:11:25,422 [main] org.jetel.graph.Phase:     FOREACH_ADDRESS_ADDRESSES ...OK
DEBUG 2015-07-03 00:11:25,423 [main] org.jetel.graph.Phase:     FOREACH_ADDRESS_ADDRESSES1 ...OK
DEBUG 2015-07-03 00:11:25,424 [main] org.jetel.graph.Phase:     FOREACH_ELECTRONIC_CONTACT_ELECTRONIC_CONTACTS ...OK
DEBUG 2015-07-03 00:11:25,425 [main] org.jetel.graph.Phase:     FOREACH_NS0_STUDENT_LIST ...OK
DEBUG 2015-07-03 00:11:25,425 [main] org.jetel.graph.Phase:     FOREACH_TELEPHONE_NUMBER_TELEPHONE_NUMBERS ...OK
DEBUG 2015-07-03 00:11:25,427 [main] org.jetel.graph.Phase:     FOREACH_TELEPHONE_NUMBER_TELEPHONE_NUMBERS1 ...OK
DEBUG 2015-07-03 00:11:25,481 [main] org.jetel.graph.Phase:     XML_READER0 ...OK
INFO  2015-07-03 00:11:25,482 [main] org.jetel.graph.Phase: Phase 0 initialized successfully.

Answer 1

Try to increase the pipe size in DataMapper( where you have enabled streaming, could see another tab called 'Pipe Size'). Default it is '2048' , you can increase pipe size and check .