All 3 NSX Managers are down following an EAM agent event

Products

• VMware NSX

Issue/Introduction

• Following an issue with EAM agent VMs. It was noticed that EAM agent was only available for few ESX hosts.
• This caused DRS to migrate all VMs to these few hosts. This led to a severe resource contention causing NSX Managers to go unresponsive. 
• The only way to recover from the resource constraint was to set DRS to "manual" and power-off then vMotion the managers to other hosts.
• After this event the NSX Managers were presenting the following error when trying to access the UI.

  • Some components of the device are not working correctly.
    Component health: MANAGER:UNKNOWN, SEARCH:UNKNOWN, UI:UP, NODE_MGMT:UP.
    Error code: 101

    
    
• The NSX Managers are on a VLAN backed NSX Segment.

Environment

VMware NSX

Cause

• Since the Managers are on a VLAN backed NSX segment, the ESX hosts were unable to get the VIF attachment for these Manager VMs from control plane. 

Resolution

• Move the NSX Managers to non NSX backed DV port group.
• Once the NSX Cluster health stabilizes, you may move the managers back to the original vLAN backed NSX Segments.
  • You can monitor the health of the NSX Manager cluster by running the command "get cluster status"

Additional Information

• Powering off ESX agent VM cause all the regular VMs to migrate out of the host in a DRS enabled cluster.