NSX Manager Alarm "Heartbeating between NSX management node and host <host-UUID> is down"

Products

• VMware NSX

Issue/Introduction

• Host Transport nodes indicate NSX configuration of "Host Disconnected"
• Attempts to resolve Host Transport node Alarm "Host Disconnected" fail and remain in a disconnected state
• Ping attempts from Host transport node to NSX Manager IP are successful
• From Host Transport node CLI, using netcat utility to test connectivity to NSX Manager port 1234 and 1235 may fail

# nc -zv <NSX_Manager_IP/FQDN> 1234

# nc -zv <NSX_Manager_IP/FQDN> 1235

• NSX Manager is reporting several expired certificates.  At least 1 of the expired certificates is APH-TN or APH-AR

Environment

• NSX v4.2+
  • For versions susceptible to JDK bug: please review the following KB article to verify NSX Manager is not being impacted by JDK bug at NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks Due to ctl Field Release Count (RC) Overflow

Cause

• Expired APH-TN and/or APH-AR certificate is preventing Host Transport nodes from re-connecting to the NSX Managers on port 1234 or 1235

Resolution

• Download carr-1.x.tar.gz script to 1 NSX Manager and follow guidance in the following KB to fix expired Host Transport node, NSX Manager certificates, and any discovered certificate mismatches Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX
•  Return to NSX "Host disconnected" Alarm (System --> Fabric --> Hosts) and Resolve the alarm by clicking Host Disconnected link
  • Resolving the Alarm will trigger NSX to restablish the connection from the Host Transport node to NSX Manager and complete NSX configuration
  • If the Resolve workflow fails, manually force a resync of the APH certificates from the TN host as root with the shell commands:
    • nsxcli -c sync-aph-certificates <Manager hostname-or-IP> username admin thumbprint <thumbprint from NSX UI > System > Appliances > View Details>
    • /etc/init.d/nsx-proxy restart

Additional Information

Please refer to the following KB article for additional troubleshooting Host Transport node certificates and additional heartbeating failures

• Alarm For Transport Node Certificate Has Expired.
• Heartbeating between NSX management node and host &lt;host-UUID&gt; is down.