ESXi hosts in Failed state following upgrade to NSX 3.2.2 or 4.0.1.1

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T 3.2.2 or NSX 4.0.1.1
• ESXi 7 Update 3
• NSX Security Only configured 
• ESXi host upgrade to 3.2.2/4.0.1.1 is reported as Success but the ESXi Transport Node is in a Failed state under System -> Fabric -> Hosts
• The Failed state is reported as:

Host configuration: NSX enabled switches already exist on host. Please run 'del nsx' command on host. Later,try the ResyncTransportNode API

• After exiting maintenance mode, VMs cannot be vMotioned to this host
• On ESXi host logs, /var/run/log/nsx-syslog
  2022-11-14T07:51:23.650Z nsx-opsagent[20278045]: NSX 20278045 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="20278454" level="ERROR" errorCode="MPA44422"] NSX enabled switches already exist on host. Pleaserun 'del nsx' command on host. Later, try the ResyncTransportNode API

Environment

VMware NSX 4.0.0.1
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

Versions 3.2.2 and 4.0.1.1 introduced a new vDS property “com.vmware.nsx.vdsSecurity.enabled” which is applied to vDS on hosts installed for Security Only deployments.
This vDS property was not set on previous NSX versions.
In an NSX upgrade scenario, the property is not applied to the vDS. When the service checks for the setting on the host vDS and doesn't find it, it believes this host has a Network and Security installation and throws this error “NSX enabled switches already exist on host”.
New installations of 3.2.2/4.0.1.1 are not impacted.

Resolution

This is a known issue impacting NSX 3.2.2 and 4.0.1.1, there is currently no resolution

Workaround:

For ESXi hosts that are in a Failed state post upgrade:

Place the ESXi host in vSphere maintenance mode.

ssh to the ESXi host
Confirm the advanced setting is not applied
#net-dvs -l | grep com.vmware.nsx.vdsSecurity.enabled
<no output returned>

Set the property on all vDS used by NSX
#net-dvs  -s com.vmware.nsx.vdsSecurity.enabled -p hostPropList <vDS name>

Reboot the ESXi host for the change to take effect, either from the vSphere client or from command-line
#reboot

Exit the host from maintenance mode.


Preventative fix prior to upgrading
ssh to each ESXi host

Set the property on all vDS used by NSX
#net-dvs  -s com.vmware.nsx.vdsSecurity.enabled -p hostPropList <vDS name>

This configuration will not impact running VMs.

Proceed with NSX upgrade as normal
 

Alternative Preventative fix prior to upgrading
Use in-place host upgrade mode instead of maintenance mode upgrade.
Hosts will show a Failed state after upgrade and vmotion will be blocked, there will be no impact on running VMs.
The hosts will change back to a Success state once the NSX Manager upgrade completes.

Please see more details on in-place host upgrade in the Upgrade Guide:
3.2.2: Operational Impact of the NSX-T Data Center Upgrade
4.0.1.1: Operational Impact of the NSX-T Data Center Upgrade