vNIC gets disconnected randomly on virtual machines in NSX environment

Products

• VMware NSX

Issue/Introduction

• VMs are connected to NSX portgroups
• VMs lose network connectivity, affecting operations and potentially causing downtime.
• vmkernel.log show logs similar to
  WARNING: NetDVS: 2777: Failed to write critical property com.vmware.net.port.fc.enabled on port abcd4f-#####-#####-###acb123f, return :Out of memory.
swsec: SwSecPortPropEnableCleanup:174: [nsx@6876 comp="nsx-esx" subcomp="swsec-20541214"]Prop swsec.enable cleanup on port 0x##3d
swsec: SwSecPortPropDiscoveryCleanup:303: [nsx@6876 comp="nsx-esx" subcomp="swsec-20541214"]Prop swsec.discovery cleanup on port 0x##3d
swsec: SwSecPortPropDiscoveryIpv6Cleanup:424: [nsx@6876 comp="nsx-esx" subcomp="swsec-20541214"]Prop swsec.discovery.ipv6 cleanup on port 0x##3d
WARNING: NetPort: 1568: failed to enable port, portID: 03##, status: Out of memory

Environment

• NSX-T 3.2.2 or earlier
• NSX 4.1.0 or earlier

Cause

Due to a memory leak in the NSX Flow-Cache module, VMs may experience network connectivity issues. This occurs either after the VM is powered on or following a vMotion to another host, where the NIC goes into disconnected state.

Resolution

This issue is fixed in NSX 3.2.3, 4.1.1 and later NSX releases

Workaround: 

1. Disable and re-enable FC on the ESXi to workaround this issue. 
   nsxdp-cli fc <disable|enable>
   
   
2. Or disable FC persistently on the ESXi
   edit /etc/vmware/nsx/nsx-cfgAgent.xml
     
<flowCache>
  <enabled>false</enabled>  <!-- Change this from true to false -->
  <mcastEnabled>false</mcastEnabled>
</flowCache>

Then restart the nsx-cfgagent service:
   /etc/init.d/nsx-cfgagent restart    

NOTE: If the workaround is run while impacted VMs are already present on the host, the network adapter of the VMs must be disconnected and reconnected to bring back the VMs into operational state.

Additional Information

To check flow cache check follow the below steps.
#] vsish
cd  /system/fastslab/fastslabs/

Pick the one that is similar to FC_InvalEntitySlab

get status

Example output:
/system/fastslab/fastslabs/FC_InvalEntitySlab-0x43#####31a##0/> get status      
       FastSlab status {       
          name:FC_InvalEntitySlab       
          size of the object:72       
          allocated size:128       
          offset of the pointer storage in the item:0       
          alignment of object:64       
          minimum number of objects:8192      
          maximum number of objects:131072      
          objects per magazine:256       
          pages of VA space per buffer:0      
          current number of allocated objects:131072      
          high water mark of allocated objects:0       
          current number of objects in buffers:131095       
          Current number of allocated buffers:9       
          Current number of allocated pages:4097       
          Physical Contiguity: 1 -> Any Physical Contiguity       
          Module that created this slab:82       
       }