NSX 4.2.1.4 upgrade failed with error "Heart beating between NSX management node and host UUID is down"

Products

• VMware NSX

Issue/Introduction

• NSX upgrade fails, and one or more ESXi hosts appear as "Disconnected" in the NSX UI.
• Error messages reported during the upgrade include:
  • "Heartbeating between NSX management node and host [HostUUID] is down."
  • "Unexpected error while upgrading upgrade unit: Command IsHostInMaintenanceMode failed on host ([HostFQDN], [HostUUID])."
• Subsequent attempts to re-prepare these hosts for NSX consistently fail at 48% completion.
• During re-preparation, the manager connectivity status for the affected hosts is displayed as "Down."

Below is the screenshot of manager connectivity and configuration state of ESXi from NSX UI

Environment

VMware NSX

Resolution

• Manually address and resolve any existing host state issues.
  • Ensure all new NSX VIBs are successfully installed on the affected ESXi host(s). Use the command esxcli software vib list | grep -i nsx to check.
  • In the NSX UI, navigate to the upgrade progress page. Click "Resolve" next to the failed upgrade unit. In the "Resolve Error" dialog box, select the relevant error checkbox and click "Resolved."
• Remove the host from NSX maintenance mode within the NSX UI.
• Re-sync the host with the NSX Manager upgrade page. Return to the NSX upgrade page and select "Retry." An initial error might still be displayed, refresh the page to obtain the latest status of the host(s) and the upgrade process.

Additional Information

If the above steps in this KB do not resolve the issue, raise a support ticket with Broadcom support selecting NSX as the product. 

Handling Log Bundles for offline review with Broadcom support.

• Collect Support Bundles for Troubleshooting VMware NSX
• Uploading files to cases on the Broadcom Support Portal
• Creating and managing Broadcom support cases