Bad network performance observed with DVS on esxi host with nsxv vibs installed or stale nsxv-dvs properties

Products

• VMware vSphere ESXi

Issue/Introduction

VMs in ESXi host may experience bad network performance after migration from NSXV to DVS during following conditions

• The VMs on the ESXi host are connected with DVS
• NSX-v vibs are either present or removed from the ESXi host successfully. 
• Stale NSX-v properties are still present on the DVS 
• If the VMs vnics are still attached with DFW (distributed firewall) of NSX-v 

There could be below messages in vmkernel logs, whenever there is a reconfiguration on the VMs vnics or if there is a re-configuration on DFW to which the vnics are attached 

####-##-##T##:##:##.###Z cpu44:#######)ixgben: ixgben_CheckTxHang:1982: vmnicxx: false hang detected on TX queue 1
####-##-##T##:##:##.###Z cpu31:#######)NetPort: 1810: disabled port <dv-port-id>  
####-##-##T##:##:##.###Z cpu31:#######)Vmxnet3: 14113: indLROPktToGuest: 1, vcd->umkShared->vrrsSelected: 4 port <dv-port-id>  
####-##-##T##:##:##.###Z cpu31:#######)Vmxnet3: 14380: Using default queue delivery for vmxnet3 for port <dv-port-id>  
####-##-##T##:##:##.###Z cpu31:#######)Vmxnet3: 1727: UPT feature is disabled.  
####-##-##T##:##:##.###Z cpu31:#######)NetPort: 3054: resuming traffic on DV port <ID>  
####-##-##T##:##:##.###Z cpu31:#######)NetPort: 1543: enabled port <dv-port-id> with mac <Mac-address of VM's vnic>  
####-##-##T##:##:##.###Z cpu31:#######)cswitch: L2Sec_EnforcePortCompliance:209: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]client <VM-name.ethx> requested promiscuous mode on port <dv-port-id>, disallowed by vswitch policy  
####-##-##T##:##:##.###Z cpu31:#######)cswitch: L2Sec_EnforcePortCompliance:209: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]client <VM-name.ethx> requested promiscuous mode on port <dv-port-id>, disallowed by vswitch policy  
####-##-##T##:##:##.###Z cpu31:#######)cswitch: L2Sec_EnforcePortCompliance:209: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]client <VM-name.ethx> requested promiscuous mode on port <dv-port-id>, disallowed by vswitch policy

Environment

ESXi 6.7

NSX-v 6.x 

Cause

Sometimes the vnics of VMs could still be attached with DFW (distributed firewall) after migrating VMs from NSXv to normal DVS which could lead to packet drops during any VM reconfiguration.

Also if ESXi host is unprepared for NSXv, sometimes few opaque properties of NSXv DVS may not be removed completely as part of removal process.

Resolution

Uninstall NSXv or remove NSXv vibs from the ESX host if the host has to be un-prepared from NSXv and if all the VMs are moved out of NSX and connected only to DVS. 

Once NSXv vibs are removed from ESXi hosts, reboot the host and ensure there are no NSXv vibs present and stale NSX properties present on DVS before onboarding any VM.

To identify the presence of stale NSXv properties on DVS even after NSXv removal from ESXi host and to take further actions to purge these properties refer below KBs

KB 385958

KB 387393

 

Additional Information

If the issue still persists, check if the symptoms observed are similar and validate/resolve as mentioned in the  as mentioned in KB 413864.