CODEWITHSUNDEEP
javaspring-bootjava-8microservicesdocker-swarm
Subh
Subh

Reputation: 414

Spring boot micro services consuming lot of memory in docker-swarm

I have some docker swarm containers running on an Ubuntu 16.04.4 LTS instance on Azure. The containers are running Java Spring Boot and Netflix OSS Applications like Eureka, Ribbon, Gateway etc. applications. I obseverd my container is taking huge size of memory although services are just REST Endpoint.

I tried to limit the memory consumption by Passing Java VM args like below, but this didn't help the size didn't get changed even after.

Please note below configuration that I am using here,

Java Version : Java 8 Alpine

Kernel Version: 4.15.0-1023-azure

Operating System: Ubuntu 16.04.4 LTS

OSType: linux

Architecture: x86_64

CPUs: 32

Total Memory: 125.9GiB

Memory footprint after docker stats

docker stats

Java VM Arguments,

docker service create --name xxxxxx-service --replicas 1 --network overnet 127.0.0.1:5000/xxxxxx-service --env JAVA_OPTS="-Xms16m -Xmx32m -XX:MaxMetaspaceSize=48m -XX:CompressedClassSpaceSize=8m -Xss256k -Xmn8m -XX:InitialCodeCacheSize=4m -XX:ReservedCodeCacheSize=8m -XX:MaxDirectMemorySize=16m -XX:+UseCGroupMemoryLimitForHeap -XX:-ShrinkHeapInSteps -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=70"

I've tried to look at the application log files within each of the containers also but can't find any memory related errors. I also tried to limit container resources. But that also didn't work for me.

Limit a container's resources

Any clue how I can troubleshoot this heavy memory Issue?

Upvotes: 2

Views: 1484

Answers (3)

yerqueri
yerqueri

Reputation: 75

You can try enabling actuator and compare the memory consumption values with the values generated by docker stats.

to enable actuator you could add the following dependency in your pom.xml file. 

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

I generally use HAL-browser for monitoring the application and consuming the actuator endpoints. you can add that using the following maven dependency.

<dependency>
    <groupId>org.springframework.data</groupId>
    <artifactId>spring-data-rest-hal-browser</artifactId>
</dependency>

In the HAL browser you could try consuming the /metrics endpoint for your application.

A sample output would look like this.

{
    "mem" : 193024,
    "mem.free" : 87693,
    "processors" : 4,
    "instance.uptime" : 305027,
    "uptime" : 307077,
    "systemload.average" : 0.11,
    "heap.committed" : 193024,
    "heap.init" : 124928,
    "heap.used" : 105330,
    "heap" : 1764352,
    "threads.peak" : 22,
    "threads.daemon" : 19,
    "threads" : 22,
    "classes" : 5819,
    "classes.loaded" : 5819,
    "classes.unloaded" : 0,
    "gc.ps_scavenge.count" : 7,
    "gc.ps_scavenge.time" : 54,
    "gc.ps_marksweep.count" : 1,
    "gc.ps_marksweep.time" : 44,
    "httpsessions.max" : -1,
    "httpsessions.active" : 0,
    "counter.status.200.root" : 1,
    "gauge.response.root" : 37.0
}

this way you can monitor the memory performance of your application and find out how much memory your application is actually consuming. If this is analogous with the report generated by docker than this is a issue with your code.

However I must state that use of actuator is not production friendly as it has a significant resource overhead in itself.

Upvotes: 1

PowerStat
PowerStat

Reputation: 3819

Another way to find out more I have used in the past is to use AspectJ's load time weaving to add special code that adds memory information to your log files. This will also slow down your system, but when your aspects have been weel written not so much as using a profile.

If possible profiling would be prefered - if not AspectJ load time weaving might become helpful.

Upvotes: 1

Andreas Hartmann
Andreas Hartmann

Reputation: 1883

You can troubleshoot this by using a profiler such as visualvm or jprofiler, they will show you where the memory is allocated (which types of objects etc.).

You shouldn't use it on a production system though, if possible, because profiling can be very CPU heavy.

Upvotes: 1

Related Questions