How to avoid memory leaks in Mule Applications?

Question

Are there some special things that must be considered to avoid memory leaks in Mule Applications?

How can we avoid memory leaks in Mule Applications?

For example; Do we actually have to remove flow variables? What must be done done explicitly by the developers of the Mule Applications and what is done (automatically) by the Mule Runtime and the JVM GC?

Answer 1

A good way to get to the memory leak suspects would be to take a heap dump (of all the nodes) right after you start seeing a decline in memory reclaim post-major GC. There are multiple tools available that help analyze the memory leaks.

There is a great blog post in the topic. This has summarized some memory-leak related issues like the following findings for example:

Finding: Pooled memory manager generally grabs 10% of JVM heap and lives with it without releasing. Fix: Switch the Grizzly Memory Manager implementation HeapMemoryManager. Note that HeapMemoryManager is the default implementation and is recommended by Grizzly for performance; albeit, Mule treats PoolMemoryManager implementation as the default.

Wrapper.conf changes:

wrapper.java.additional.<XX>=-Dorg.glassfish.grizzly.DEFAULT_MEMORY_MANAGER=org.glassfish.grizzly.memory.HeapMemoryManager

Finding: Async logging was being used widely and associated Log4J was observed to be holding a lot of JVM memory. The default setting of 256*1024 slots was apparently too high. Since this RingBuffer does not grow or shrink, a high fixed size with each slot allocated as a separate object (RingBufferLogEvent), each holding a log event, could occupy a considerable amount of memory.

Fix: Reduce the Log4J RingBuffer size to 128 in wrapper.conf or log4j2.xml

wrapper.java.additional.<XX>=-DAsyncLoggerConfig.RingBufferSize=128

Or, in log4j2.xml:

<AsyncLogger name="DebugLog" level="info" includeLocation="true" ringBufferSize="128">

Memory leak due to default HazelCast implementation used for aggregator components (splitter-Aggregator pattern).

Finding: Heap analysis pointed memory being held up by default HazelCast objectstore implementation used in splitter-aggregator components used in specific flows. It appeared as if the store was not getting expired appropriately.

Fix: Custom Object store implementation (subclass of PartitionedInMemoryObjectStore) was written and TTL (TimeToLive) for entries explicitly defined.

@Override 
public void expire(int entryTTL, int maxEntries, String partitionName) throws ObjectStoreException
{
    super.expire(entryTTL, maxEntries, partitionName);
    if (getPrivatePartitionSize(partitionName) == 0) {
disposePartition(partitionName);
    }
}

reference : https://dzone.com/articles/enduring-black-fridays-with-mulesoft-apis

Answer 2

General Recommendations

• Session Variables

For applications with many endpoints, prefer fewer and smaller session variables over many or large ones.The session scope is serialized and deserialized every time a message crosses an endpoint, even a VM endpoint. So if an application has a lot of endpoints, it will involve many serializations/deserializations. Using fewer and smaller session variables helps minimize this overhead.

• Payload format

When it comes to performance, not all formats are equal. Some payload formats allow faster access to data than others. Bean payloads tend to be the fastest for Mule applications. So if it is a viable option given other considerations, create payloads in Java objects.

• Data Extraction

Mule Expression Language (MEL) can be used to extract data from messages.8 In terms of performance, allowing MEL to extract data can be preferable to using a scripting language. Scripting languages are dynamically typed. Some are even interpreted at runtime. Those factors can generate overhead that may degrade performance.

• Flow References

Flow references are a pleasantly direct way to enable flow communication within an application. Flow references are preferred for communications between flows than VM endpoints. Flow references inject messages into the target flow without intermediate steps. Although the VM connector is an in-memory protocol. It emulates transport semantics that serialize and deserialize parts of messages. The phenomenon is especially notable in the Session scope. As such, flow references are superior to VM endpoints for the purpose of inter-flow communication because the former avoid unnecessary overhead generated by serialization and deserialization.

JVM and GC flags can be set for Mule in wrapper.conf .

It is easy to get passionate about a particular Java Virtual Machine (JVM) or garbage collection (GC) method. JRockit versus HotSpot, parallel mark-and-sweep (MS) versus G1.

• MuleSoft uses HotSpot, the standard Oracle JVM. HotSpot is well-supported and easy to customize for a variety of purposes. MuleSoft’s performance testing emphasizes throughput, and thus parallel GC. HotSpot is also easy to optimize for response time. The tips in the following sections show how to calibrate HotSpot for throughput or response time.

• Designate the initial and maximum heap sizes to be the same value. This can be done by setting MaxMetaspaceSize=MetaspaceSize and MaxNewSize=NewSize . Doing so can avoid the need for the JVM to dynamically allocate additional memory during runtime. The flags are set in wrapper.conf .

  e.g. wrapper.java.additional.16=-XX:NewSize=1365m wrapper.java.additional.17=-XX:MaxNewSize=1365m wrapper.java.additional.18=-XX:MetaspaceSize=256m wrapper.java.additional.19=-XX:MaxMetaspaceSize=256m wrapper.java.additional.20=-Xms=2048m wrapper.java.additional.21=-Xmx=2048m

• There are at least two reasons why such dynamic reallocation can hinder performance. First, the JVM performs a major GC for each heap resize. A full GC stops all threads for a period of time. That holds even when using concurrent mark-and-sweep (CMS). World-stopping should always be minimized, other things being equal. This is especially crucial for applications prioritizing low response times. Dynamic heap resizing creates a second worry when memory is tight. Suppose the JVM increases its heap size during runtime and the system does not have enough free memory pages readily available. As a result, some pages for a kernel-chosen process might be swapped out to disk. The circumstance would incur slowdown due to increased disk IO.

Garbage Collection HotSpot is equipped with three canonical garbage collection (GC) mechanisms. These are serial, parallel, and concurrent mark-and-sweep (CMS).18 Garbage First (G1) has recently been added to the list.19 The JVM uses parallel GC by default on machines with 2 or more physical processors and 2 or more GB of physical memory.

Parallel GC is the default garbage collection algorithm in HotSpot JVM. When triggered, it uses multiple threads to scan, move and collect the unreachable objects in the heap. CMS GC (Concurrent-Mark-Sweep) Concurrent mark-and-sweep (CMS) GC is designed to reduce application pauses by running most of the cleaning phases concurrently with the application threads, hence it offers more control over the stall time which affects the application response time. Here is an example demonstrating how to set the JVM to use CMS, plus other options. Set the following in Mule’s wrapper.conf file. Section 6, “Sample Configuration Files” gives additional context in which the flags are set.

wrapper.java.additional.22=-XX:+UseConcMarkSweepGC
wrapper.java.additional.23=-XX:CMSInitiatingOccupancyFraction=65
wrapper.java.additional.24=-XX:UseCMSInitiatingOccupancyOnly

The flag -XX:CMSInitiatingOccupancyFraction designates a percentage of the total heap usage. When that percentage is reached, the JVM will trigger a CMS GC. A value of 40 to 70 typically suffices for applications running on Mule. If the value is too low, it could result in excessive, premature collections. It is usually recommend to start at a relatively higher value for -XX:CMSInitiatingOccupancyFraction and decrease it as needed to optimize for the fewest CMS events for the best performance. Specify -XX:+UseCMSInitiatingOccupancyOnly when designating -XX: +CMSInitiatingOccupancyFraction . Otherwise, the JVM attempts to dynamically adjust the value for -XX:+CMSInitiatingOccupancyFraction. A changing value is undesirable in most production scenarios. That is because dynamic adjustment is based on statistical analysis that may not reliably account for load spikes.

GC logging is a good idea for performance tests. The GC log, once enabled, provides extremely valuable information about the activities in the heap and how they affect the runtime performance. GC logging tends to have little overhead for disk IO. Here is an example of how to enable various aspects of GC logging. Add these configurations to Mule’s wrapper.conf file.

wrapper.java.additional.4=-XX:+PrintGCApplicationStoppedTime
wrapper.java.additional.5=-XX:+PrintGCDetails
wrapper.java.additional.6=-XX:+PrintGCDateStamps
wrapper.java.additional.7=-XX:+PrintTenuringDistribution
wrapper.java.additional.8=-XX:ErrorFile=%MULE_HOME%/logs/err.log
wrapper.java.additional.9=-Xloggc:%MULE_HOME%/logs/gc.log
wrapper.java.additional.10=-XX:+HeapDumpOnOutOfMemoryError