NSX Manager Upgrade Fails with Error Code '36580'

Products

• VMware NSX

Issue/Introduction

When attempting an upgrade of the NSX Management Cluster, the upgrade fails on one or more nodes. The NSX Manager User Interface or upgrade logs display the following error message:

Unexpected error while upgrading upgrade unit: [MPP] Node upgrade failed : org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: "{"error_code": 36580, "error_message": "Error proxying request to: ####-####-####-####.", "module_name": "node-services"}".


• Severe CPU throttling or high utilization spikes on the affected NSX Manager appliance.
• The application service frequently restarts, showing Disconnected from cluster or Disconnected from database inside the proton logs.
• Inter-node network statistics (ping.stats) show average Round Trip Time (RTT) latency values climbing well above 10ms, frequently spiking past 100ms.
• Active vCenter Server alarms indicating physical NIC (pNIC) errors or performance degradation on the underlying storage layer (e.g., vSAN Datastore) on ESXi host where the NSX VM was residing.

Environment

VMware NSX

Cause

The issue is caused by physical network infrastructure degradation on the ESXi host hosting the failing NSX Manager appliance.

Because the NSX Management Cluster requires an inter-node network latency of under 10ms to maintain distributed cluster state sync, the combination of storage and network drops causes the node-services module to time out, drop its database connection, and fail the upgrade unit transaction.

Resolution

• Log in to the vSphere Client and identify the ESXi host currently execution-hosting the degraded NSX Manager appliance.
• Use vMotion to migrate the affected NSX Manager virtual machine to a healthy ESXi host within the cluster that is free of physical NIC errors and storage alarms.
• Log in to the command-line interface (CLI) of the affected NSX Manager appliance as admin and execute the top command to verify that CPU utilization has normalized and services have stabilized.
• Log back in to the NSX Manager User Interface, navigate to the Upgrade tab, and resume the upgrade workflow.

Additional Information

For detailed documentation on maximum clustering latencies and architecture limits, see the VMware Cloud Foundation Technical Documentation.