VMs unable to migrate on to NSX upgraded ESXi hosts by DRS/Manually

Products

• VMware NSX

Issue/Introduction

• On NSX UI, Upgrade window you see the below warning message for ESXi hosts after the NSX upgrade.

Connection between host <UUID> and NSX Controller is UNKNOWN. Response: Client is responding to heartbeats.

• ESXi hosts NSX Configuration shows as Failed in NSX UI.
• vMotion to these ESXi hosts fails at 19%.
• Checking /var/run/log/nsxaVim.log of ESXi host failing NSX installation/configuration should show logs similar to below: 

YYYY-MM-DDT14:48:09Z nsxaVim: [2102356]: INFO Entered update lockdown exception to [add] user [nsx-user]^@
YYYY-MM-DDT14:48:09Z nsxaVim: [2102356]: INFO Adding user nsx-user in lockdown exception list^@
YYYY-MM-DDT14:48:09Z nsxaVim: [2102356]: WARNING User <user name> does not exist retrying updating exception list^@  


Environment

VMware NSX 
VMware NSX-T Datacenter

Cause

If the HostClient or vCenter Lockdown exception list includes Active Directory users which are subsequently removed from the AD domain server, the ESXi host will not automatically remove the user from the list of lockdown exceptions. 
This "stale" Lockdown mode exception user can cause the nsxaApp service to go down on the ESXi host, which in turn will prevent the Host from successfully being configured as NSX transport node.

Resolution

• Remove the user account, which is mentioned in the log /var/run/log/nsxaVim.log, as above, from the lockdown exception list.
  
  • Select the ESXi Host in VC
  • Navigate to Configure>Security Profile>Lockdown Mode
    
  • Select Edit
  • Select Exception Users
  • Click the 3 dots next to the user throwing the "does not exist" error in /var/run/log/nsxdavim.log

  • Remove User
• After removing the user restart the nsx-opsagent service using this command:  /etc/init.d/nsx-opsagent restart
• Add the user back to the exception list if required
• Refresh NSX UI ESXi Transport Nodes section, you see the NSX Configuration status changed to Success.

Note: The user may also exist in the HostClient UI, please review and remove the mentioned user from the HostClient UI if it exists. 

Additional Information

Please refer to the KB Article - Unable to prepare ESXi hosts for NSX due to existing stale Lockdown Mode exception user(s) for similar issue.