CODEWITHSUNDEEP
apachegarbage-collectionout-of-memoryapache-camel
chand
chand

Reputation: 11

Facing java.lang.OutOfMemoryError:GC overhead limit exceeded when using Camel Aggregator

I am facing a java.lang.OutOfMemoryError: GC overhead limit exceeded for a file which is generated using aggreagation. Below is my route

    <route id="sendToFileRoute">
        <from uri="direct:sendToFile" />
        <choice>
            <when>
                <!-- check to ignore if from LIVEFEED -->
                <simple>${property.FileName} != 'LIVEFEED'</simple>
                <to uri="bean:intermediateFileProcessor?method=processSNE" />
                <camel:aggregate completionSize="300" strategyRef="aggrMessToFile" forceCompletionOnStop="true" completionTimeout="5000" >
                    <camel:correlationExpression>
                        <camel:simple>${property.stageFileName}</camel:simple>
                    </camel:correlationExpression>
                <camel:to uri="file://{{intermediate.dir.path}}?fileExist=Append&amp;fileName=$simple{property.stageFileName}" />
                </camel:aggregate>
            </when>
        </choice>
    </route>

The file generated is something like - V2_10.246.84.69_INI_TIMER_US_20150528_102000514_32.xml

Below is the error log

2015:05:28:10:43:09.552 [Camel (locationContext) thread #2 - file:///cert/pubsub/location/mount/] ERROR com.walmart.eim.location.common.handle
rs.ExceptionHandler RequestId  -  2845692472690849  :  FileName  -  V2_10.246.84.69_INI_TIMER_US_20150528_102000514_32.xml -
2015:05:28:10:43:37.107 [Camel (locationContext) thread #7 - timer://deltaTimer] ERROR com.walmart.eim.location.common.handlers.ExceptionHandl
er  -
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[Message: [Body is null]]
        at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1360)
        at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:272)
        at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:88)
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
         at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:139)
        at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:64)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2015:05:28:10:43:43.307 [Camel (locationContext) thread #2 - file:///cert/pubsub/location/mount/] ERROR com.walmart.eim.location.common.handle
rs.ExceptionHandler RequestId  -  2845692472690849  :  FileName  -  V2_10.246.84.69_INI_TIMER_US_20150528_102000514_32.xml -
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[V3ConfidES_10.246.84.69_INITIAL_TIMER_
US_20150528_102000514_32.xml]
        at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1360)
        at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:272)
        at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:224)
        at org.apache.camel.processor.Splitter.process(Splitter.java:98)
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:352)
        at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:199)
        at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:165)
        at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187)
        at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2015:05:28:10:43:43.353 [Camel (locationContext) thread #7 - timer://deltaTimer] ERROR com.walmart.eim.location.common.handlers.ExceptionHandl
er RequestId  -  2846346936275989 -
java.lang.NullPointerException
        at org.apache.velocity.runtime.parser.node.ASTIdentifier.execute(ASTIdentifier.java:148)
        at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
        at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
        at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
        at org.apache.velocity.runtime.RuntimeInstance.render(RuntimeInstance.java:1378)
        at org.apache.velocity.runtime.RuntimeInstance.evaluate(RuntimeInstance.java:1314)
        at org.apache.velocity.app.VelocityEngine.evaluate(VelocityEngine.java:272)
        at org.apache.camel.component.velocity.VelocityEndpoint.onExchange(VelocityEndpoint.java:191)
        at org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71)
        at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
        at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110)
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:103)
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:56)
        at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:110)
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
 at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        at org.apache.camel.processor.FatalFallbackErrorHandler.process(FatalFallbackErrorHandler.java:42)
        at org.apache.camel.processor.RedeliveryErrorHandler.deliverToFailureProcessor(RedeliveryErrorHandler.java:839)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:337)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
        at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:139)
        at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:64)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

Can some one please help me on resolving this?

Upvotes: 1

Views: 1448

Answers (1)

Ashoka
Ashoka

Reputation: 943

The Camel Aggregrator2 is running of memory possibly due to the large dataset. In this case it's better to make use of the database based aggregation strategy using -- org.apache.camel.processor.aggregate.jdbc.JdbcAggregationRepository

This will be a more scalable option. Here's a great article : http://labs.bsb.com/2011/04/jdbc-persistence-for-camel-aggregator/

Upvotes: 1

Related Questions