ESXi Transport Nodes are disconnected in NSX UI due to expired certificates

Products

• VMware NSX

Issue/Introduction

Within the NSX Manager UI, the clusters of transport nodes are reported to have a preparation failure, NSX configuration reports the host is disconnected, and the status of the host is unknown:

Possible alarms that can be triggered by this issue are, though it is not an exhaustive list:

• Management Channel To Transport Node Down
• Heartbeating between NSX Management node and host <host UUID> is down

 

Environment

VMware NSX 4.x

Cause

While existing data traffic typically continues to flow, the expiration of internal certificates (specifically those used for Transport Nodes) disrupts the secure communication channel between the ESXi hosts and the NSX Management Plane. When this happens, the host’s status in the NSX Manager UI often transitions to a "Failed" or "Disconnected" state, which blocks management-level operations like vMotion.

Resolution

To resolve this issue the expired certificates will need to be replaced.

• Self-signed certificates can be replaced by running the CARR script on an NSX manager
  • Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX
  • Please note that SSH root access is needed to successfully upload the script to an NSX Manager
    • Enable ssh root access for NSX appliances
• CA signed certificates will need to be replaced by generating a new CA-signed certificate.
  • Replace NSX Manager Certificates Using CA-Signed Certificates in NSX 4.x

Once the expired certificates have been replaced the NSX configuration can be triggered again:

1. Log into the NSX UI using the admin credentials
2. Click on System > Fabric > Hosts  and expand the cluster with the disconnected hosts. 
3. Under the column NSX Configuration click on the text Host Disconnected
4. If there are still errors shown here, select the check box next to the error and select Resolve
   • Another pop-up will appear and ask again if the error should be resolved, click Resolve again.
5. Repeat steps 1 through 5 for all other hosts that have the disconnected issue. 
6. The NSX Configuration for the ESXi host should immediately begin on its own and the hosts should eventually complete the configuration and report a status a successful NSX configuration with UP under the column Node Status.

Additional Information

If the hosts are disconnected in NSX but not due to an expired certificate please refer to the following articles:

• Heartbeating between NSX management node and host is down.
• ESXi Host's NSX Configuration status shows Host Disconnected in NSX UI
• NSX manager UI is showing esxi host transport node in 'Host Disconnected' state

If the issue is not resolved, please open a case with VMware by Broadcom support and include the following data:

• ESXi host logs
• NSX Manager logs
• Screenshots of any errors seen
• Any troubleshooting done previously

For assistance opening a support case, please review Creating and managing Broadcom support request (SR) cases.