NSX Manager High CPU and Degraded Performance due to Proton Thread Leak when onboarding AVI

Products

• VMware NSX
• VMware Avi Load Balancer

Issue/Introduction

• NSX Manager node(s) frequently enter a Degraded state.
• Slow Web UI response and hanging CLI commands.
• High CPU utilization observed for the Java process (user uproton).
• CPU Load average on the NSX Manager is very high. 
• NSX Manager logs (/var/log/proton/nsxapi.log) show entries similar to: java.lang.OutOfMemoryError: unable to create native thread.

[Timestamp] ERROR l3-tasks1 TaskUtils$LoggingErrorHandler 6356 Unexpected error occurred in scheduled task
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Unknown Source) ~[?:?]

[Timestamp] ERROR org.corfudb.runtime.collections.streaming.StreamPollingScheduler-worker-2 StreamingTask 6356 StreamingTask: encountered exception java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached during client notification callback, listener: com.vmware.nsx.management.proton.commonagent.common.impl.CertificateChangeNotificationServiceImpl@2c86a7b9 name listener_com.vmware.nsx.management.proton.commonagent.common.impl.CertificateChangeNotificationServiceImpl@2c86a7b9_nsx_common_agent id ########-c1fc-3411-936e-############

[Timestamp]  WARN ShaHeartbeater VcfGrpcClient 6356 MONITORING [nsx@4413 comp="nsx-manager" level="WARNING" s2comp="sha-vcf-grpc-client" subc
omp="manager"] Failed in sending heartbeat:process_name: "manager"

io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) ~[?:?]
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249) ~[?:?]
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167) ~[?:?]
        at vmware.nsx.sha.ShaClientServiceGrpc$ShaClientServiceBlockingStub.sendHeartbeat(ShaClientServiceGrpc.java:231) ~[?:?]
        at com.vmware.nsx.sha.VcfGrpcClient.sendHeartbeat(VcfGrpcClient.java:236) ~[libnsx_sha_client.jar:?]
        at com.vmware.nsx.sha.VcfGrpcServiceImpl.lambda$startHeartbeat$4(VcfGrpcServiceImpl.java:751) ~[libnsx_sha_client.jar:?]
        at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[?:?]
        at io.grpc.internal.ManagedChannelImpl$ExecutorHolder.execute(ManagedChannelImpl.java:2151) ~[?:?]
        at io.grpc.internal.DnsNameResolver.resolve(DnsNameResolver.java:389) ~[?:?]
        at io.grpc.internal.DnsNameResolver.refresh(DnsNameResolver.java:212) ~[?:?]
        at io.grpc.internal.ForwardingNameResolver.refresh(ForwardingNameResolver.java:58) ~[?:?]
        at io.grpc.internal.ManagedChannelImpl.refreshNameResolution(ManagedChannelImpl.java:467) ~[?:?]
        at io.grpc.internal.ManagedChannelImpl.access$4800(ManagedChannelImpl.java:124) ~[?:?]
        at io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1LoadBalancerRefreshNameResolution.run(ManagedChannelImpl.java:1416) ~[?:?]
        at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:94) ~[?:?]
        at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:126) ~[?:?]

• NSX manager logs (/var/log/proton/proton-tomcat-wrapper.log)

INFO   | jvm 1    | [Timestamp] |    java.lang.Thread.State: TIMED_WAITING (sleeping)
INFO   | jvm 1    | [Timestamp] |       at java.lang.Thread.sleep(java.base@17.0.17/Native Method)
INFO   | jvm 1    | [Timestamp] |       at com.vmware.nsx.management.policy.paceagent.services.GrpcCommunicationServiceImpl.pollAndSendMessage(GrpcCommunicationServiceImpl.java:191)
INFO   | jvm 1    | [Timestamp] |       at com.vmware.nsx.management.policy.paceagent.services.GrpcCommunicationServiceImpl$$Lambda$3583/0x0000############.run(Unknown Source)
INFO   | jvm 1    | [Timestamp] |       at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.17/Unknown Source)
INFO   | jvm 1    | [Timestamp] |       at java.util.concurrent.FutureTask.run(java.base@17.0.17/Unknown Source)
INFO   | jvm 1    | [Timestamp] |       at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.17/Unknown Source)
INFO   | jvm 1    | [Timestamp] |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.17/Unknown Source)
INFO   | jvm 1    | [Timestamp] |       at java.lang.Thread.run(java.base@17.0.17/Unknown Source)

• This issue is specific to environments where AVI is onboarded to an NSX Manager instance. When onboarded, the PACE agent creates a GrpcClient to connect and transmit data to AVI.

Environment

VMware NSX 9.x
VMware Avi Loadbalancer

Cause

This issue is caused by a native thread leak within the NSX Manager's Proton service in which GrpcCommunicationServiceImpl.pollAndSendMessage fails to properly reclaim threads, leading to an accumulation of thousands of orphaned threads. Once the operating system's native thread limit is reached, the JVM can no longer spawn new threads, causing the service to become unresponsive or crash.

Resolution

This issue is resolved in future releases of VMware NSX.

Workaround:

• To temporarily restore functionality and clear the leaked threads, perform a restart of proton service on the affected NSX Manager node(s).

Warning: Restarting the proton service will result in a temporary loss of access to the NSX Management Plane for that node.

Log into the NSX Manager root mode. Execute the following command:

systemctl restart proton

Verify the service status:

systemctl status proton