Performance degradation in Web layer after upgrade to Spring 6

[alexey-anufriev]

Affects: 6.0.10 (Spring Boot 3.1.1)

Summary:
After upgrade to Spring Boot 3.x application started to take more time for web requests processing.

Description:
Initially, the application was based on Spring Boot 2.7.14 (Java 17). But since we upgraded to Spring Boot 3.1.1 we noticed performance degradation in web layer. Requests processing took extra time in comparison with Boot 2.x. Upgrade to Spring Boot 3.1.2 made it ever worse. Tomcat's MessageBytes serialization started to perform even slower. So we reverted to Spring Boot 3.1.1.

One more thing that was discovered in the profiler is slow performance of AntMatcher. So we followed the recommendation from upgrade guide of Spring Boot 2.6 and witched to PathParser. As it turns out, this did not make much of an improvement (see details below).

CPU

Here are some details captured using DataDog profiler.

Here is a request processing of the URL that has path variables:

And here is a request processing of the URL that has no path variables:

Memory

One more thing that was discovered is memory consumption, most of the allocations are happening here:

This is the size per minute

---

How can this be improved? One thought is around some kind of caching for PathParser to avoid extra calculations while request matching.

[bclozel]

Thanks for the report @alexey-anufriev , but there not so much actionable here, or at least conflicting messages.

You're first mentioning Tomcat's MessageBytes performance, which is out of our control.
You are also pointing to AntPathMatcher and PathPattern as areas of improvements. At this point, I'm not sure where the performance loss comes from and whether this is related to those classes. Did you compare the profiling output of 2.7.x and 3.1.x for your application? Did the ratio of the request matching phase grow in the overall execution with this upgrade?

Again, without much context, it's hard to understand why this part of the execution is slow. Request matching is of course an important part and we've optimized it with PathPatternParser and PathPattern. Depending on the request and if the path matches, some other conditions are processed (content negotiation, request parameters and others). It's not surprising some requests require less processing for the matching phase as optimizations happen along the way.

Discussing about performance is often tricky as it's easy to waste time setting up an imperfect benchmark test harness and draw the wrong conclusions. Comparing the same application with two different versions and pointing to the difference should be a good first step. Pointing to parts of the request processing and asking for performance improvements is not a good way to start; we've got multiple micro-benchmarks in the Framework codebase and spending time there is not always the most efficient way to optimize for your case.

How can this be improved? One thought is around some kind of caching for PathParser to avoid extra calculations while request matching.

The PathPatternParser is only called once during startup and create efficient PathPattern instances for matching against incoming requests. We optimized this quite a bit already in the past using a JMH benchmark. I don't understand the suggestion here. What should be cached?

[alexey-anufriev]

Fair enough, let me prepare benchmarks.

[bclozel]

@alexey-anufriev I'm not suggesting you to prepare benchmarks, quite the opposite. It takes a lot of time to design and implement a benchmark and consider its biases.

Here, I'm really requesting a full picture of the CPU time (your datadog flame graph) for the 2.7.x and 3.1.x versions of the same application. Comparing CPU time there should point to possible regressions that I could look into.

[alexey-anufriev]

Ok, let me try to do fast comparison.

This is before the upgrade:

And this is after:

[bclozel]

Thanks for the update!

If I understand correctly, both applications are Spring MVC, one using the AntPathMatcher and the other the PathPatternParser variant. It would be nice to check whether the same behavior can be seen in Spring Boot 2.7. By default there, the PathPatternParser strategy is chosen as well. Did you opt-out explicitly in 2.7.x? If so, can you try with the same matching strategy in 2.7.x?

it looks like the MvcRequestMatcher is triggering the matching operation here, as you've probably migrated your Security configuration as requested by Spring Security. Still, the matching phase should not take longer than usual so this bit is interesting.

It would be interesting to get more information about the particular endpoints involved here:

• can you share the patterns exposed by your controllers?
• can you share the request you are sending to the application?

With that information I can start building a sample application involving Spring Security, a few controller endpoints and find out if there is a performance issue with the matching phase.