VM's on NSX lose network connectivity after vMotion during NSX upgrade of ESXi transport nodes

Products

• VMware NSX

Issue/Introduction

• While performing the ESXi transport node NSX upgrade from NSX version before 4.1.1 to version 4.1.1 or later , VM's migrating between hosts may lose network connectivity after migration. 
• The network connectivity loss will affect all network connectivity to the affected VM. 
• ESXi host or VM reboots do not clear the problem. 
• After SSH'ing to the ESXi host running a VM affected by this issue and running command 'net-stats -l', the "com.vmware.vswitch.port.swsec.discovery.vmtools" property is observed above property "com.vmware.vswitch.port.swsec.enabled = true".

                com.vmware.vswitch.port.swsec.discovery.vmtools = 0x 1. 0
                        propType = POLICY
                com.vmware.port.opaque.network.id = 0ab4b507-xxxx-xxxx-xxxx-7040cb0c6b34 ,      propType = RUNTIME
                com.vmware.port.opaque.network.type = nsx.LogicalSwitch ,       propType = RUNTIME
                com.vmware.vswitch.port.swsec.enabled = true ,  propType = POL

• In the ESXi host's vmkernel logs, these log prints are observed. 

2024-12-01T20:12:40.674Z cpu56:9634951)swsec: SwSecPortPropDiscoveryVmToolsWrite:489: [nsx@6876 comp="nsx-esx" subcomp="swsec-23786738"]Prop swsec.discovery.tools write on port 0x800xxxx, len: 2
2024-12-01T20:12:40.674Z cpu56:9634951)WARNING: swsec: SwSecPortPropDiscoveryVmToolsWrite:507: [nsx@6876 comp="nsx-esx" subcomp="swsec-23786738"]SwSec filter not found for port 0x800xxxx
2024-12-01T20:12:40.674Z cpu56:9634951)WARNING: NetPort: 1551: failed to enable port, portID: 0x800xxxx, status: Not found

Environment

VMware NSX 4.1.1 and later

VMware NSX-T Datacenter 3.x

Cause

• The "com.vmware.vswitch.port.swsec.discovery.vmtools" property is new to VMware NSX 4.1.1 and is not present in earlier versions.
• During the NSX upgrade of a cluster of ESXi transport nodes, if a VM is migrated from an ESXi host with NSX 3.x installed to an ESXi host with NSX 4.1.1 or later installed, the new property "com.vmware.vswitch.port.swsec.discovery.vmtools" will be added to it.
• These properties can become out of order with the new property "com.vmware.vswitch.port.swsec.discovery.vmtools" being evaluated before swsec is enabled by property "com.vmware.vswitch.port.swsec.enabled = true", resulting in this failure and the VM port becoming blocked. 

The following sequence of events scenario can lead to this issue;

• A VM was created on host with NSX version earlier than 4.1.1 and does not include property com.vmware.vswitch.port.swsec.discovery.vmtools.
• The VM is migrated to an upgraded NSX version 4.1.1 or later host and picks up the new property com.vmware.vswitch.port.swsec.discovery.vmtools.
• The VM migrates back to an older NSX version (under NSX version 4.1.1) host.
• The VM is power cycled which will cause the properties to become out of order with com.vmware.vswitch.port.swsec.discovery.vmtools ahead of com.vmware.vswitch.port.swsec.enabled = true as seen in the output of 'net-dvs -l'.
• The VM is migrated back to an upgrade ESXi host with NSX version 4.1.1 or later.  

Resolution

This is a known issue impacting VMware NSX and will be fixed in a future version.

Workaround:

• For an already affected VM, the order of properties can be reset by changing the VM's configured portgroup to any other portgroup.
• The issue can be avoided by ensuring VM's are not vMotioned or power cycled while the host cluster is still on mixed NSX versions.
• The "In-place" NSX upgrade option will also help avoid this issue by limiting the vMotions occurring during the ESXi host transport node upgrade process, as well as the time the ESXi hosts remain on mixed versions.  

The documentation for NSX "In-place" upgrades can be found here, at Step 5; "Configuring and Upgrading Hosts".

Additional Information

For additional information see Troubleshooting NSX Network Connectivity Issues.