How to measure benefit of virtual thread?

Question

I migrated a Java 17 application to Java 21 and enabled virtual thread for the application.

Microservice Detail:

• 20 requests per second on a container.
• Scaling in switched on for this service. Whenever average CPU hits 30% new container is launched.
• Usually 50 containers are running for this service.
• Without virtual thread, one container used to have an average of 2.5k platform threads running every minute.
• We are using GRPC server on this service.
• Request Detail:
  • One requests which is on platform thread, generally uses 6-7 parallel platform thread to fetch data from different data source(I/O).
  • We have converted the later 6-7 platform threads to virtual thread using ExecutorService.
  • We are not able to move the user request to virtual thread as grcp server does not support it.

When we go live, I can see earlier system used to have 2.5k active platform thread on an avergage. Now it has reduced to 200. But apart from this I am not observing any other benefit.

What I thought, I will observe

• Increase in throughput per container.
• Number of containers used should have reduced as now less context switching because of less active platform threads.

I am seeing no benefit apart from reducing number of platform threads.

What I am missing? What other parameters I should check which will show improvements?

Answer 1

At the end, it's up to you what you consider a (significant) improvement. If something is important for you, you should measure it. If your application performs better (according to metrics important for you) with virtual threads, use them. If virtual threads make your application perform worse, don't use them (Note that in some cases, your metrics also include things like application performance in high-pressure scenarios which may be different).

What to measure

For example, if the platform threads are using a lot of memory, you might need less memory with virtual threads. Given that, you might want to measure the amount of memory you are using in your application However, if you are serving more requests at the same time (that may take longer because some other part of the system is limiting) with virtual threads, the amount of memory may also increase because you need more memory for more requests.

This brings us to another metric you could measure: How many requests are you concurrently serving (with and without virtual threads)? Maybe you are handling more requests at the same time with virtual threads (e.g. you might be in the case where you handle 50 concurrent requests each taking 50ms with platform threads and handle 100 concurrent requests each taking 100ms with virtual threads - this may be the case if something else is limiting throughput and take more memory). You might want to measure both the average/median/peak/etc. time it takes for a request as well as the amount of requests you are serving in parallel.

Are there other things important for you? Measure them! And also measure the differences between platform and virtual threads.

Thoughts on throughput

You mentioned that you would have expected throughput improvements when switching your virtual threads but didn't see that. Maybe another component of your system is limiting throughput.
For example, if you are limiting the amount of grpc/user threads that are available (or if you are limiting the amount of requests served at the same time), you won't get higher throughput. Your previous ways of limiting concurrent requests may be reasonable with your previous approach but they would stop you from increasing throughput by using virtual threads. Since you are now using less platform threads overall, you could increase the number of platform threads you are using for handling user/grpc requests (or making your concurrency limitations less strict).
Another thing that could limit your throughput is limitations on I/O. If you are limited by the throughput of your disk, adding more operations requiring disk accesses (which could be what you are doing with virtual threads by allowing to handle more requests) won't improve your throughput.
If you are expecting better throughput, you should investigate what actually limits throughput (both with virtual and platform threads).

Virtual threads may improve CPU utilization. But that doesn't help you much if your CPU utilization is already maximized. If you are hitting your 30% CPU bound with platform threads, you will probably still hit it with virtual threads.

Other than that, you'll likely not get significant improvements with just a few virtual threads. If you want virtual threads to actually improve performance, you should probably have at least a thousand of them. Check how many threads you actually have in your applications and whether it's worth to convert them to virtual threads. After that, also check how many virtual threads you have.

Other things to try

Apart from that, you could try other things.
For example, you are giving the virtual threads names using your ThreadFactory. If you don't need these names, you could skip this step which may result in less memory being used in case you have a lot of virtual threads.
Another thing worth trying, especially if you are limited by the platform threads your grpc server is using, is updating that grpc server. Maybe a newer version supporting virtual threads available?