VMs are violating vmHost affinity where NSX-T Data Center is deployed - InsufficientAgentVmsDeployed

Products

• VMware NSX Networking
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Where affinity rules are applied, DRS is not moving the virtual machines to the correct hosts
• Seen with vCenter 7.0.3 in an NSX-T environment
• Affinity rules are respected with Manual vMotion.
• In /var/log/vmware/vpxd/vpxd.log on the vCenter appliance you may encounter entries similar to

info vpxd[07474] [Originator@6876 sub=drmLogger opID=SWI-5c73cfb3] 1 VMs are violating vmHost affinity(true)/antiAffinity(false) rule vm_rule_temp
info vpxd[07474] [Originator@6876 sub=cdrsPlmt opID=SWI-5c73cfb3] Vm [vim.VirtualMachine:vm-name,tes_DRS] failed constraint check false on host [vim.HostSystem:host-1,esx01.corp.local] with <obj xm
lns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:vim25" versionId="7.0.2.0" xsi:type="LocalizedMethodFault"><fault xsi:type="InsufficientAgentVmsDeployed"><hostName>esx01.corp.local</hostName><requiredNumAgentVms>1</requiredNumAgentVms><currentNumAgentVms>0</currentNumAgentVms></fault><localizedMessage></localizedMessage></obj>

• In /var/log/cm-inventory/cm-inventory.log on the NSX-T Manager you may encounter entries similar to

INFO http-nio-127.0.0.1-7443-exec-15 CmInventoryFacadeImpl 15417 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Retrieved cm config info from cm plugin instance, cmPluginStatusData= CmPluginStatusData{id=xxxxxxxx-2486-4007-a4cd-xxxxxxxxxxxx, server=vcenter.corp.local, cmPluginStatus=CmPluginStatusInfo{status=FAILED, cmConnectionStatus=DOWN, errors=[{"moduleName":"cm-inventory","errorCode":40107,"errorMessage":"Unable to connect to Compute Manager vcenter.corp.local. Please edit compute manager details if FQDN or thumbprint is changed. If the issue persists, please check whether the https port 443 and http port 80 are open in the firewall on all NSX nodes."}, {"moduleName":"cm-inventory","errorCode":40118,"errorMessage":"Compute Manager vcenter.corp.local can not be connected, as its thumbprint does not match. Please edit compute manager details if thumbprint is changed."}]}}

Environment

VMware vSphere ESXi 7.0
VMware NSX-T Data Center

Cause

Communication between NSX-T Manager and vCenter is down.

Resolution

If a new certificate has been applied to vCenter, ensure it is applied in the correct format: Root > Intermediate > Server/Leaf. 
Update the Compute Manager (vCenter) certificate thumbprint in NSX to establish connection between NSX-T and vCenter.

Use the following command on the vCenter cli shell to get the certificate thumbprint if it is not auto populated in NSX-T

openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256

Recreate DRS rules.
Review KB 90086 and installation guide for details on updating a compute manager thumbprint.

 

Additional Information

Impact/Risks:

NSX-T is not acknowledging agents that are stopped into hook states. NSX is unable to fetch EAM status from EAM service or provide callbacks.