Unexpected error while upgrading upgrade unit: [MPP] Node upgrade failed" (Error Code 36580) during NSX Manager upgrade

Products

• VMware NSX

Issue/Introduction

NSX Manager upgrade fails on a specific manager node within the management cluster.

The /var/log/upgrade-coordinator/upgrade-coordinator.log records the following error:

Unexpected error while upgrading upgrade unit: [MPP] Node upgrade failed : org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 Internal Server Error: "{"error_code": 36580, "error_message": "Error proxying request to: <uuid>.", "module_name": "node-services"}"

The impacted manager node generates the following alarms indicating DNS resolution failures:

• DNS lookup failed for Manager node <uuid> with FQDN <>.
• Reverse DNS lookup failed for Manager node <uuid> with IP address <>.

Environment

VMware NSX

Cause

DNS resolution failure prevents the upgrade coordinator from proxying requests to the target manager node. This occurs due to incorrect or missing DNS entries (A or PTR records) for the NSX Manager, or general DNS server configuration errors and unavailability.

Resolution

Validate and correct the DNS resolution for the affected NSX Manager node.

1. Log in to the affected NSX Manager appliance via SSH using the root account.
2. Verify the configured DNS servers on the appliance by reading the resolv.conf file: cat /etc/resolv.conf
3. Verify forward DNS lookup (FQDN to IP resolution) using the nslookup utility: nslookup <FQDN> Example output should return the correct IP address.
4. Verify reverse DNS lookup (IP to FQDN resolution) to ensure the PTR record is correct: nslookup <IP_Address> Example output should return the correct FQDN.
5. Alternatively, use the dig command for more detailed diagnostic output:
• Forward: dig <FQDN>
• Reverse: dig -x <IP_Address>