NSX Manager Alarm "Heartbeating between NSX management node and host <host-UUID> is down"

Products

• VMware NSX

Issue/Introduction

• Host Transport nodes indicate NSX configuration of "Host Disconnected"
• Attempts to resolve Host Transport node Alarm "Host Disconnected" fail and remain in a disconnected state
• Ping attempts from Host transport node to NSX Manager IP are successful
• From Host Transport node CLI, using netcat utility to test connectivity to NSX Manager port 1234 snd 1235 fail

# nc -zv <NSX_Manager_IP/FQDN> 1234

# nc -zv <NSX_Manager_IP/FQDN> 1235

• NSX Manager is reporting several expired certificate.  1 of the expired certificates is APH-TN 

Environment

• NSX Version: 4.2.1.3
  • This version is also susceptible to JDK bug.  Please review the following KB article to verify NSX Manager is not being impacted by JDK bug:  NSX is Impacted by JDK-8330017: ForkJoinPool Stops Executing Tasks Due to ctl Field Release Count (RC) Overflow

Cause

• Expired APH-TN certificate preventing Host Transport nodes from re-connecting to the NSX Managers on port 1234 and 1235

Resolution

• Download carr-1.x.tar.gz script to 1 NSX Manager and follow guidance in the following KB to fix expired Host Transport node, NSX Manager certificates, and any discovered certificate mismatches Using Certificate Analyzer, Results and Recovery (CARR) Script to fix certificate related issues in NSX
•  Return to NSX "Host disconnected" Alarm (System --> Fabric --> Hosts) and Resolve the alarm by clicking Host Disconnected link
  • Resolving the Alarm will trigger NSX to restablish the connection from the Host Transport node to NSX Manager and complete NSX configuration

Additional Information

Please refer to the following KB article for additional troubleshooting Host Transport node certificates and additional heartbeating failures

• Alarm For Transport Node Certificate Has Expired.
• Heartbeating between NSX management node and host &lt;host-UUID&gt; is down.