The vDS portgroup link goes down post NSX V removal

Products

• VMware NSX Data Center for vSphere

Issue/Introduction

• Un-preparing the NSX V Transport Node with NSX V vibs being removed successfully
• Post VIB removal, the ports assigned to VMs in DVPG show down and network connectivity to the VMs using the port group is impacted.
• hostd logs show the port going in a disconnected state as below:

2024-11-26T13:33:53.713Z info hostd[2100066] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=m3y7xkgj-37298-auto-ss3-h5:########-bd-01-6d-####] Event 309 : The dvPort ### was disconnected in the vSphere Distributed Switch in ha-datacenter.

• vmkernel logs show that certain the port in question is still looking for NSX properties like "dvfilter-generic-vmware-swsec" and when it is not able to find the property the port is marked as disconnected as below :
  
  2024-11-26T13:32:06.881Z cpu18:2102736 opID=58793961)Net: 3007: associated dvPort ### with portID 0x#######
  2024-11-26T13:32:06.881Z cpu18:2102736 opID=58793961)Vmxnet3: 14062: indLROPktToGuest: 1, vcd->umkShared->vrrsSelected: 6 port 0x#######
  2024-11-26T13:32:06.881Z cpu18:2102736 opID=58793961)Vmxnet3: 14329: Using default queue delivery for vmxnet3 for port 0x#######
  2024-11-26T13:32:06.881Z cpu18:2102736 opID=58793961)Vmxnet3: 1727: UPT feature is disabled.
  2024-11-26T13:32:06.881Z cpu18:2102736 opID=58793961)NetPort: 3054: resuming traffic on DV port ###
  2024-11-26T13:32:06.882Z cpu18:2102736 opID=58793961)NetPort: 1543: enabled port 0x####### with mac 00:50:56:##:##:##
  2024-11-26T13:32:06.882Z cpu2:2097683)DVFilter: 4128: Could not find filter 'dvfilter-generic-vmware-swsec' <<<<<<<<<<<<<<<<<<<
  2024-11-26T13:32:06.882Z cpu2:2097683)cswitch: VSwitchDisablePTIOChainRemoveCB:1288: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Remove ptDisable IOChain Handle for port 0x####### <<<<<<<<<<<
  2024-11-26T13:32:06.882Z cpu2:2097683)netschedHClk: NetSchedHClkPortQuiesce:7153: vmnic3: received a force quiesce for port 0x#######
  2024-11-26T13:32:06.882Z cpu2:2097683)netschedHClk: NetSchedHClkPortQuiesce:7153: vmnic4: received a force quiesce for port 0x#######
  2024-11-26T13:32:06.882Z cpu2:2097683)NetPort: 1810: disabled port 0x####### <<<<<<<<<<
  2024-11-26T13:32:06.882Z cpu2:2097683)Net: 3608: dissociate dvPort ### from port 0x####### <<<<<<<<
  2024-11-26T13:32:06.882Z cpu2:2097683)Net: 3614: disconnected client from port 0x#######<<<<<<<<

 

• net-dvs -l from the host in question shows that the port in still has NSX-based properties even when the NSX vibs are successfully removed from the ESXi host.
  
  
  "dvfilter-generic-vmware-swsec.config"
   "com.vmware.common.port.dvfilteraltvmx"

 

• Login to vCenter DB and run below query: (to check for any NSX opaque property data for the ports in question.)

         /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

         select * from vpx_dvs_opaque_data;"

Environment

VMware NSX Data Center for vSphere 6.4.x

Cause

Few opaque properties of NSX V were not removed as part of NSX V removal process.

This led to ports still looking for the values in the property even though there is no NSX in the transport Node

Resolution

Below are the workarounds to mitigate the issue.

Workaround (A):

• Create a port group/VLAN and move the impacted VM to the new Port Group/VLAN.

Workaround (B):

• If the number of affected ports is more in numbers, then open an SR with Broadcom Support for cleanup using scripts.