Rare multi-step failure can lead to loss of communication between an NSX Manager and a host

Products

• VMware NSX for vSphere

Issue/Introduction

Symptoms:
There is a disconnection between the NSX Manager and a host.

Environment

VMware NSX for vSphere 6.3.x

Resolution

This is a known issue affecting VMware NSX for vSphere 6.3.6.

Currently, there is no resolution.

Workaround:
To work around this issue,  use a synchronize API on the NSX Manager to force an update of communication channel credentials between NSX Manager and the failed host.

For example:

https://NSXMGR_IP/api/2.0/nwfabric/configure?action=synchronize
  
<nwFabricFeatureConfig>
    <featureId>com.vmware.vshield.vsm.messagingInfra</featureId>
    <resourceConfig>
        <resourceId>host-52</resourceId>
    </resourceConfig>
</nwFabricFeatureConfig>

Additional Information

One rare example of how a host disconnection may occur without the possibility of recovery are the following:

• The network between the NSX Manager and a host partitions resulting in heartbeat failure between the two.
• NSX Manager attempts to push new credentials to the host through the vCenter Server, believing that the cause of the connection break is de-synchronization of credentials. But at that instant, the connection between the vCenter Server and the host also happens to be down. 
• When the connection between NSX Manager and the host heals, another attempt to re-establish communication by pushing new credentials occurs. However, this process also fails.
• At this point, any attempts to reset the credentials for the host will no longer occur.