NSX - Host connectivity intermittently shows status 'Unknown'

Products

• VMware NSX

Issue/Introduction

• ESXi host reports "UNKNOWN" status (grey) in NSX UI intermittently before reverting to UP and green
• The associated error message while disconnected is "Heartbeating between NSX management node and host <UUID> is down."
• NSX version is 4.1 or 4.2
• The Compute Manager (vCenter) certificate expired and was replaced prior to the issue.
• NSX manager syslog (log path: /var/log/syslog)shows connectivity issues to the Compute Manager's inventory matching with status DOWN for the host TNs:
  
  WARN inventoryTasksScheduler2 CmInventoryService 75514 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="cm-inventory"] CM plugin status not found for <CM_UUID>, plugin may not be running on this node.
INFO http-nio-127.0.0.1-7443-exec-4 MpRestService 75514 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="########-####-####-####-############" subcomp="cm-inventory" username="<username>"] Getting certificate using REST API nsxapi/api/v1/trust-management/certificates/?action=get_private
INFO http-nio-127.0.0.1-7443-exec-4 LoggingRestTemplate 75514 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" reqId="########-####-####-####-############" subcomp="cm-inventory" username="<username>"] Request::URI:http://localhost:7440/nsxapi/api/v1/trust-management/certificates/?action=get_private method:GET
ERROR http-nio-127.0.0.1-7443-exec-4 MpInventoryRestClient 75514 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP40304" level="ERROR" reqId="########-####-####-####-############" subcomp="cm-inventory" username="<username>"] Error in rest call. url= /nsxapi/api/v1/trust-management/certificates/?action=get_private , method= GET , response= {
  "httpStatus" : "BAD_REQUEST",
  "error_code" : 268,
  "module_name" : "common-services",
  "error_message" : "Request has unknown parameters: action."
} , error= 400 : "{<EOL>  "httpStatus" : "BAD_REQUEST",<EOL>  "error_code" : 268,<EOL>  "module_name" : "common-services",<EOL>  "error_message" : "Request has unknown parameters: action."<EOL>}"
org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : "{<EOL>  "httpStatus" : "BAD_REQUEST",<EOL>  "error_code" : 268,<EOL>  "module_name" : "common-services",<EOL>  "error_message" : "Request has unknown parameters:
action."<EOL>}"

Environment

VMware NSX

Cause

The NSX Controllers are failing to validate the new Compute Manager (CM) certificate.

Resolution

Perform one of the following workarounds:

• Perform a rolling reboot of all the NSX Manager nodes to reestablish the connectivity with the CM inventory.
  or
• Restart the CM service in all NSX Manager nodes:
  restart service cm-inventory