Spring app crashes with CPU utilization and memory errors

Products

• VMware Tanzu Application Service

Issue/Introduction

We are getting cpu utilization and out of memory issue for application deployed in Tanzu foundation

2025-12-04T01:10:50.47+0530 [APP/PROC/WEB/8] OUT {"timestamp":"2025-12-03T14:40:50.46981466-05:00","severity":"WARN","service":"example-app-name","thread":"ExampleThreadName","class":"o.s.b.a.d.r.RedisReactiveHealthIndicator","springProfile":"EnvironmentName","methodName":"?","lineNo":"?","message":"sypher:[syftokenized] Health check failed","stacktrace":"java.lang.OutOfMemoryError: Cannot reserve 8388608 bytes of direct buffer memory (allocated: 5524046, limit: 10485760)\n\tat
java.base/java.nio.Bits.reserveMemory(Unknown Source)\n\tat
java.base/java.nio.DirectByteBuffer.<init>(Unknown Source)\n\tat java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source)\n\tat io.netty.buffer.UnpooledDirectByteBuf.allocateDirect(UnpooledDirectByteBuf.java:104)\n\tat io.netty.buffer.UnpooledDirectByteBuf.capacity(UnpooledDirectByteBuf.java:156)\n\tat io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:305)\n\tat io.netty.buffer.AbstractBy

Resolution

Since the error showed that there was not enough available direct memory within the defined maximum, we increased the MaxDirectMemorySize value.

cf set-env <MY_APP> JAVA_OPTS '-XX:MaxDirectMemorySize=128M'

We restaged the app, then checked and verified that the new value of MaxDirectMemorySize was in effect. Production traffic was then redirected while we tailed the app (cf log syf-chat-gpt-service-green). No errors were observed; all application instances took traffic and remained at low-reasonable CPU utilization levels.

NOTE: Manually overriding memory calculator settings by MaxDirectMemorySize can potentially degrade application performance and functionality if not set carefully. Use caution and reduce or revert values if adverse results are observed.

Additional Information

Because there was not enough information to make a determination as to the root cause, we suggest that customer:

1. increase Reactor Netty's leak detection level to "advanced" or "paranoid" i.e.
   
   • spring.netty.leak-detection=paranoid
2. raise the logging level to "debug" to obtain detailed stack traces and connection IDs
   •  logging.level.reactor.netty=DEBUG
     
     

Reactor Netty threw the error we observed. These log levels will provide additional details that can help identify the components interacting with the leaked ByteBuf.

Revert MaxDirectMemorySize to the original value and try to reproduce the issue. Change settings on a non-production environment and run a load test. Open a new ticket if you need further assistance.