Goodbye Performance Issues: How Project Loom Eliminates Asynchrony

Anyone who develops a backend application with more than a handful of users knows that most performance problems are related to I/O. In modern web applications, these are typically network calls. Whether it’s REST requests to another service, queries to an external database, or communication with middleware. We handle all these cases - consciously or unconsciously - in asynchronous threads. This way, we no longer block the main thread. Doing this correctly is not trivial. Project Loom will be released in September with Java 21. Virtual threads will enable us to write our programs entirely sequentially. And yet, we are still making optimal use of our resources. But what opportunities does this open up for the Spring ecosystem and how will it affect us in the future?

What happens when a REST request is sent to the Spring backend

The embedded web server - by default Tomcat - continuously reads the configured port for HTTP requests. As soon as a new request is identified in the stream, Tomcat delegates the processing of the data to a separate thread. In a standard Spring application, these are the “http-nio” threads you find in your stacktrace. These threads are often referred to as request threads. Within these threads, depending on the metadata supplied, such as the path or authentication, various filters of the servlet FilterChain are executed. In the course of this, the request ultimately ends up in the controller method, where the logic of the application begins. At this point, we are still within the request thread.

Blocking I/O operations prevent the processing of new requests

Suppose it is necessary in our business logic to send an HTTP request to an external service. We need the result to process the actual request of the user. If we execute this synchronously, for example with RestTemplate, we block the request thread. By default, Tomcat provides 10 threads for processing. This means that no more than 10 requests can be processed in parallel at any one time. This becomes a problem when we block the threads as described above with an operation.

We unblock our request threads through asynchronous processing

The usual solution to this problem is the introduction of new threads waiting for the response of the blocking operation. In Spring, depending on the situation, there are many different mechanisms for this. At its core, the idea is the same. Instead of executing the blocking operation, such as the call of an external API, on the request thread, we initialize a new thread. In most cases, these threads are managed in their own thread pools.

It’s not easy to keep track of your system

In a typical backend application, this behavior extends the logic across multiple thread boundaries. And this is no trivial task. It must be ensured that thread-crossing data are included. The SecurityContext is an example from the Spring Framework. But also monitoring relevant data, such as a TraceId, must be passed on to the other processing threads. In addition, some I/O operations are essential for the logic. In these cases, it must already be taken into account at higher layers that asynchronous operations are being executed. For example, we declare in the controller that a Future is delivered, so that the web server knows that it can free up the request threads in the meantime. This way, we mix responsibilities across the layers. This is not easy. It makes our application more complex.

Optimizing the thread pool configuration is difficult

To squeeze the optimal performance out of a system, it is essential to adjust its thread pools to the requirements. Too many threads cause too many context-switches in the operating system. Too few can lead to idle times when all are in a WAITING state at the same time. How the threads behave largely depends on the logic. This becomes even more complicated when this logic can take on different shapes. And how do we deal with short-term spikes in the system? All these questions can and must be answered. But these requirements also make our system significantly more complex.

Project Loom lets us forget all of this - Synchronous all the way

And this is exactly where Project Loom will be revolutionary. With Project Loom, virtual threads are introduced. The JVM will autonomously manage the resources of the real system threads. If a virtual thread executes a blocking operation, the JVM will no longer be bothered by it. The real hardware resources can still be used perfectly. This means that we can write our entire business code synchronously. We write it as it logically makes sense, and the JVM takes care of the optimal use of the system resources. There is no disadvantage to having a paused virtual thread in the system. As soon as Tomcat supports virtual threads, the example discussed here is no longer a problem. Tomcat can create a virtual thread for each request and the JVM takes care of the sensible distribution of system resources.

Virtual threads will be released with Java 21 in September 2023. And Spring, along with the entire associated ecosystem, is working on comprehensive support by the end of the year. For every developer, this will bring a massive - positive - change. I’m looking forward to it!