Why is Tomcat 7 on 32 Bit CPU/OS/Java so much slower than on 64 Bit?

Question

                        Raspberry Pi3   Raspberry Pi3   Odroid C2       Odroid XU4
                        1,20 GHz        1,20 GHz        1,5 GHz         2,0 GHz
                        Debian 32 Bit   SuSE 64 Bit     Ubuntu 64 Bit   Ubuntu 32 Bit
Start Apache Tomcat     04:30,00        00:29,06        00:27,45        04:08,39
1. page (1. request)    00:50,00        00:03,91        00:03,66        00:24,75
1. page (2. request)    00:03,30        00:00,79        00:00,77        00:02,39

I'm working on an IoT kind of project and needed to test if some web frontend implemented in Java using Tomcat as web server is "fast enough" on our possible hardware. We need to choose between Raspberry Pi3, Odroid C2 and Odroid XU4. Pi3 and C2 both have a 64 Bit CPU with slightly different performance according to their specs, XU4 has a 32 Bit CPU only and should be faster as the other two in theory as well. The important thing is that Pi3 is by default running a 32 Bit OS even if it has a 64 Bit CPU, the XU4 is running 32 Bit as well, but the C2 is running a 64 Bit OS incl. 64 Bit Java etc.

Comparing all those devices in default settings we found that the C2 was significantly faster than the other both. It was 4+ minutes vs. ~30 seconds for a restart of Tomcat with some test application of ours. Additionally, tools like htop showed that most of the runtime all cores of the C2 were used, whereas Pi3 and XU4 were mostly only able to put one core under load. That great performance difference was the same after Tomcat has loaded and we were able to browse through our test app: It was ~1,5 seconds vs. 4 to 5,5 seconds for just browsing some page with some CSS/JS.

While the default OS for the Pi3 is 32 Bit only, we were able to successfully install a special 64 Bit SuSE distribution. And guess what happened? The performance was much closer now to what we saw on the C2 already, almost the same for many tests, even though the Pi3 is clocked at only 1,2 vs. 1,5 GHz of the C2. Especially interesting was that now all cores of the Pi3 were under load most of the time as well, so overall behaviour was very much like the C2 now.

So by only switching to 64 Bit OS, Java etc. we saw that dramatically improvement in performance. Everything else was the same, same test app, Tomcat etc., nothing overclocked, no other storage or else. How can that be? What is responsible for that dramatic improvement?

With a 64 Bit OS we see that all cores of the devices are more under load compared to 32 Bit. But why should the Linux kernel scheduler care about if it's running on 32 or 64 Bit this much?

And if it doesn't and the difference comes from Java, why/how that? Shouldn't a 32 Bit and 64 Bit JVM perform almost identically in such a simple test? Shouldn't both especially put almost the same load on cores and not behave that different? The architecture of the OS shouldn't have any effect on how many threads are used inside the JVM, that is mostly under control of Tomcat and our test app and therefore didn't change. According to what I've read about performance of 32 vs. 64 Bit Java, the difference should be negligible in my use case. Additionally, other users with a better performance of a 64 Bit JVM don't seem to have a factor of 4 to 5 like I'm seeing and the differences on CPU load of individual cores aren't explained as well.

Our test is not I/O bound, we don't allocate much memory or work with many threads or such, it's almost strictly CPU, only compiling Java classes and publishing HTML, CSS and JS. But we see very different load on the cores depending on 32/64 Bit and very different performance results.

One of my colleagues said he read somewhere that Java is internally working with 64 Bit values only and that therefore on a 32 Bit CPU/OS more cycles are needed to process the same thing. I guess his source doesn't mean really everything, but only references/pointers to memory like for objects. But I can't believe that a 32 Bit JVM is internally really using 64 Bit pointers for no reason, especially if even optimizations like compressed oops exist. But might be an explanation, so any ideas on that?

If it's of any interest, the packages on the 32 Bit OS all had "armhf" as architecture, compared to "arm64" on the 64 Bit ones. I thought that might have influence on how Java was built, maybe really using 64 Bit pointers for some weird reason?

Java was OpenJDK 8 always, same architecture like OS and as current as the package manager of the OS provides. Pi3 with SuSE had 1.8_144, UB provided 1.8_131 for both 32 Bit and 64 Bit installations, all were server VMs. Additionally, the Linux Kernel was different e.g. Pi3 with SuSE vs. C2 and XU 4 with UB: Pi3 had some current 4.x, C2 some old 3.14 and XU 4 some current 4.9 as well.

So, any ideas on where the difference comes from? Thanks!

Answer 1

You've told you installed OpenJDK 8 from the standard package.

There has never been an optimized build of OpenJDK 8 for ARM 32 (at least on Debian and Ubuntu). The default package is built from "Zero" port which does not even have a JIT compiler.

root@localhost:~# java -server -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-1~bpo8+1-b11)
OpenJDK Zero VM (build 25.131-b11, interpreted mode)
        ^^^^^^^                    ^^^^^^^^^^^^^^^^

Try to install Oracle JDK manually from Java SE downloads page.
It has an optimized HotSpot JVM inside. And it indeed works much faster.

root@localhost:~# /usr/java/jdk1.8.0_131/bin/java -server -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) Server VM (build 25.131-b11, mixed mode)
     ^^^^^^^^^^^^^^^^^^^^^                    ^^^^^^^^^^

On the contrast, Aarch64 port of HotSpot JVM has been a part of OpenJDK for a long time. So, on a 64-bit OS the default OpenJDK package comes with HotSpot JVM that includes an optimizing JIT compiler.