CODEWITHSUNDEEP
javarestspring-bootparallel-processingstream
jatingupte
jatingupte

Reputation: 11

Java Process consumes more than 100% CPU

I am running a spring-boot application on VM having Ubuntu 18.04.2. The application basically has the REST API's which gets the data from the database, massages it and sends back to the client. After running for a while, the CPU consumption of the application reaches around 190%. I checked this using top command. And when I do top -H, I can see 2 threads each consuming around 90% of the CPU like following. 

PID USER     PR NI  VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND   
2185  root   20 0 3680020 626440 17968 R 99.0  7.7  23:55.98 java                                                                                                       
28726 root   20 0 3680020 626440 17968 R 96.4  7.7  24:01.21 java

And after this, the application just stops working. I have to kill and restart the application to make it work again.

To debug the issue, I tried hitting an API from postman runner with 0-sec delay, 20 iterations, and 5 parallel requests. I could able to reproduce the issue by doing this. I took thread dump using jstack -F and observed the threads which are consuming the CPU are in IN_JAVA state. Please check the following.

Thread 2185: (state = IN_JAVA)
 - java.util.TreeMap.getEntry(java.lang.Object) @bci=79, line=359 (Compiled frame; information may be imprecise)
 - java.util.TreeMap.get(java.lang.Object) @bci=2, line=278 (Compiled frame)
 - services.impl.ChannelServiceImpl.currentSalesCategoryParam1(java.util.Map, java.lang.String, java.lang.String, java.lang.String, java.lang.String, java.lang.Double, java.lang.Double) @bci=154, line=206 (Compiled frame)
 - services.impl.ChannelServiceImpl.lambda$getChannelsVoFromSalesRecord$2(java.util.Map, models.SalesRecord) @bci=65, line=103 (Compiled frame)
 - services.impl.ChannelServiceImpl$$Lambda$13.accept(java.lang.Object) @bci=8 (Compiled frame)
 - java.util.stream.ForEachOps$ForEachOp$OfRef.accept(java.lang.Object) @bci=5, line=184 (Compiled frame)
 - java.util.ArrayList$ArrayListSpliterator.forEachRemaining(java.util.function.Consumer) @bci=99, line=1382 (Compiled frame)
 - java.util.stream.AbstractPipeline.copyInto(java.util.stream.Sink, java.util.Spliterator) @bci=32, line=481 (Compiled frame)
 - java.util.stream.ForEachOps$ForEachTask.compute() @bci=103, line=291 (Compiled frame)
 - java.util.concurrent.CountedCompleter.exec() @bci=1, line=731 (Compiled frame)
 - java.util.concurrent.ForkJoinTask.doExec() @bci=10, line=289 (Interpreted frame)
 - java.util.concurrent.ForkJoinPool$WorkQueue.runTask(java.util.concurrent.ForkJoinTask) @bci=21, line=1056 (Interpreted frame)
 - java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=35, line=1692 (Interpreted frame)
 - java.util.concurrent.ForkJoinWorkerThread.run() @bci=24, line=157 (Interpreted frame)

From my initial understanding, I think the parallel streams are the culprit, but I am not sure about it.

I am using the following code to process around 9500 of the records. The currentSalesCategoryParam1 creates a 5 level nested map for further data massaging.

        salesRecords.parallelStream().forEach(record -> {
            String levelOneKey = MappingUtils.resolveBaseChannel(record.getBuyingGroupDesc(),
                    record.getRetailChannel());
            String levelTwoKey = MappingUtils.resolveChannelName(record.getBuyingGroupDesc(),
                    record.getRetailChannel());
            String levelThreeKey = record.getBrandName();
            String levelFourKey = MappingUtils.resolveSalesType(record.getLyMeasure());
            Double currentRowVariance = MappingUtils.currentRowVariance(record.getTyValue(), record.getLyValue());
            currentSalesCategoryParam1(dataAggregationMap, levelOneKey, levelTwoKey, levelThreeKey, levelFourKey,
                    currentRowVariance, record.getLyValue());
});


private static void currentSalesCategoryParam1(
            Map<String, Map<String, Map<String, Map<String, Map<String, Double>>>>> dataAggregation, String levelOneKey,
            String levelTwoKey, String levelThreeKey, String levelFourKey, Double variance, Double lySum) {

        Map<String, Map<String, Map<String, Map<String, Double>>>> levelOneMap = dataAggregation.get(levelOneKey);
        if (CollectionUtils.isEmpty(levelOneMap)) {
            levelOneMap = new TreeMap<>();
        }

        Map<String, Map<String, Map<String, Double>>> levelTwoMap = levelOneMap.get(levelTwoKey);
        Map<String, Map<String, Map<String, Double>>> levelTwoTotalMap = levelOneMap.get("Total");
        if (CollectionUtils.isEmpty(levelTwoMap)) {
            levelTwoMap = new TreeMap<>();
        }
        if (CollectionUtils.isEmpty(levelTwoTotalMap)) {
            levelTwoTotalMap = new TreeMap<>();
        }

        Map<String, Map<String, Double>> levelThreeMap = levelTwoMap.get(levelThreeKey);
        Map<String, Map<String, Double>> levelThreeTotalMap = levelTwoTotalMap.get(levelThreeKey);
        if (CollectionUtils.isEmpty(levelThreeMap)) {
            levelThreeMap = new TreeMap<>();
        }
        if (CollectionUtils.isEmpty(levelThreeTotalMap)) {
            levelThreeTotalMap = new TreeMap<>();
        }

        Map<String, Double> levelFourMap = levelThreeMap.get(levelFourKey);
        Map<String, Double> levelFourTotalMap = levelThreeTotalMap.get(levelFourKey);
        if (CollectionUtils.isEmpty(levelFourMap)) {
            levelFourMap = new TreeMap<>();
            levelFourMap.put(MappingConstants.VARIANCE, 0.0);
            levelFourMap.put(MappingConstants.LY_SUM, 0.0);
        }
        if (CollectionUtils.isEmpty(levelFourTotalMap)) {
            levelFourTotalMap = new TreeMap<>();
            levelFourTotalMap.put(MappingConstants.VARIANCE, 0.0);
            levelFourTotalMap.put(MappingConstants.LY_SUM, 0.0);
        }

        levelFourMap.put(MappingConstants.VARIANCE, levelFourMap.get(MappingConstants.VARIANCE) + variance);
        levelFourMap.put(MappingConstants.LY_SUM, levelFourMap.get(MappingConstants.LY_SUM) + lySum);
        levelFourTotalMap.put(MappingConstants.VARIANCE, levelFourMap.get(MappingConstants.VARIANCE) + variance);
        levelFourTotalMap.put(MappingConstants.LY_SUM, levelFourMap.get(MappingConstants.LY_SUM) + lySum);
        levelThreeMap.put(levelFourKey, levelFourMap);
        levelThreeTotalMap.put(levelFourKey, levelFourTotalMap);
        levelTwoMap.put(levelThreeKey, levelThreeMap);
        levelTwoTotalMap.put(levelThreeKey, levelThreeTotalMap);
        levelOneMap.put(levelTwoKey, levelTwoMap);
        levelOneMap.put("Total", levelTwoTotalMap);
        dataAggregation.put(levelOneKey, levelOneMap);
    }```

Upvotes: 1

Views: 661

Answers (1)

joseph
joseph

Reputation: 2835

You need to sychronize or guard your access to the TreeMap or switch to ConcurrentMap. Multiple threads have concurrently modified your tree map, creating a cycle in the tree map. When your code accesses the tree map again it get stuck in a cycle. Read more about the problem in these 3 great articles:

Upvotes: 1

Related Questions