NSX-T UI shows the ESXi host status in 'unknown' state

Products

• VMware NSX

Issue/Introduction

• ESXi hosts will be in an “unknown state” in NSX-T UI.
• VMs might get disconnected when vmotioned to these problematic hosts.
• Need to check in NSX-T manager : /var/log/phonehome-coordinator/phonehome-coordinator-tomcat-wrapper.log 
  
  STATUS | wrapper | 2022/02/14 08:42:07 | Launching a JVM...
INFO | jvm 1 | 2022/02/14 08:42:07 | WrapperManager: Initializing...
INFO | jvm 1 | 2022/02/14 08:42:09 | 2022-02-14T08:42:09.672Z INFO org.apache.catalina.startup.Catalina load Initialization processed in 1574 ms
INFO | jvm 1 | 2022/02/14 09:16:08 | 2022-02-14T09:16:08.645Z INFO org.apache.catalina.startup.Catalina start Server startup in 2038971 ms

Environment

VMware NSX-T Data Center

Cause

Currently there is a hard-coded 30-minute timeout for processes to register with APH, which allows them to receive messages from hosts. So if the PHC JVM takes more than 30 minutes to initialize, it may miss this 30-minute registration window and thus not receiving any status updates from the hosts, which results in unknown status being displayed for those hosts.

Resolution

Workaround:
Restart PHC service by running the following command in NSX-T manager CLI:

"restart service telemetry"

Note: restart of telemetry service will not have any impact on datapath. 

Additional Information

Impact/Risks:
VMs getting disconnected when vmotioned to the problematic Hosts