Packet Drops are Seen when Edge connected to DVPGs is Moved to Prepped ESXi Host
search cancel

Packet Drops are Seen when Edge connected to DVPGs is Moved to Prepped ESXi Host

book

Article ID: 445209

calendar_today

Updated On:

Products

VMware NSX VMware Cloud Foundation

Issue/Introduction

When normal workload VMs are placed on the prepped ESXi host where the edge is residing, then packet drops are seen. 

Issue is seen in below configurations: 

  • When edge "Uplink profile"/"Edge Host Switch" does not have transport VLAN set OR transport VLAN(TEP VLAN) set as 0
  • Fast path(fp-ethx) interface corresponding to Edge VTEP is connected to non-trunk DVPG
  • Besides Edge VTEP interface, Edge Management Interface/Vlan HostSwitch Interface is also connected to non-trunk DVPG

Environment

VMware Cloud Foundation 9.1.0
NSX 9.1.0

Cause

  • When edge "Uplink profile"/"Overlay Host Switch" does not have transport VLAN set OR transport VLAN set as 0, then NestedTNConfig property is expected to be set only on DVPG port corresponding to VTEP interface. But in this issue, NestedTNConfig com.vmware.port.extraConfig.vdl2.nestedTNConfig = version=1;vlan=0,label=<labelID> property is set in non-VTEP edge DVPG interfaces as well e.g.
  • Management interface (non-VTEP interface)
  • Fast path(fp-ethx) interfaces corresponding to VLAN Host Switch (non-VTEP interface)

Using net-dvs -l command to check if com.vmware.port.extraConfig.vdl2.nestedTNConfig property with "0" vlan is set on DVPG ports, which are connected to above mentioned non-VTEP edge interfaces. Run below command on ESX Host on which Edge node is present. Sample net-dvs -l command output looks as below → 

============================ net-dvs -l command output =======================================      

port ##:
                com.vmware.common.port.alias = , propType = CONFIG
                com.vmware.common.port.connectid = ####, propType = CONFIG
                com.vmware.common.port.volatile.status = inUse linkUp portID=#### propType = RUNTIME <<<<< Check for port ID corresponding to non-VTEP edge interfaces

                ...
                com.vmware.port.extraConfig.vdl2.nestedTNConfig = version=1;vlan=0,label=####, propType = CONFIG <<<<< Check if nestedTNConfig property is set

 

 

 

 

Resolution

The issue has been fixed since NSX 9.1.1, available at Broadcom Downloads.
If you are having difficulty finding and downloading software, review the Download Broadcom products and software KB.

Workaround includes below mentioned 3 steps.

Note:

Maintenance window should be taken before applying workaround.

Workaround should be only applied in below mentioned scenario

Scenario -

  1. When "Uplink profile" and "Edge Overlay Host Switch", both has Transport VLAN/TEP VLAN → Not Configured OR set as "0"

  2. Edge DVPGs connected to VTEP interfaces are non-Trunk DVPGs

Workaround Steps(For 9.1.0.0 version and above Setups)

Step 1 : Update TEP VLAN in Edge Overlay HostSwitch to a non-zero value

    • Open Edge Node Uplink Mapping -> VIEW DETAILS for the impacted Edge node.
    • As shown below in highlighted part, enter the VTEP VLAN id which you want to use for Edge VTEP.
    • VTEP VLAN specified should be a non-zero value.

 Step 2 : Attach the Edge VTEP vNIC to "VLAN Trunk  DVPG" which contains the VTEP VLAN in the Trunk range

    • DVPG Configuration before Workaround will look something like below

    • If DVPG connected to edge VTEP interfaces is a non-trunk as mentioned above. Then we need to make this DVPG as "VLAN Trunk-DVPG" with below steps.  
  •  
    • Make DVPG as "VLAN Trunk-DVPG" ,
      • login to vCenter → Go to Networks → Select DVPG → Then configure VLAN trunk range for this DVPG. In VLAN trunk range just specify edge VTEP VLAN ID as mentioned below -

Connect this "VLAN Trunk-DVPG" to Edge VTEP Interface → 

    • After above configuration we need to attach this VLAN Trunk DVPG to edge VTEP interface i.e. "Uplink interfaces" from Overlay HostSwitch

Step 3 : After above 2 steps are performed, we need to disconnect and re-connect "Vlan Host Switch interfaces" and "Management interfaces"

    • In 1st two steps issue for VTEP fast path(fp-ethx) interface will be resolved

    • In this 3rd step, we are resolving issue for "Vlan Host Switch interfaces"(fp-ethx) and "Management interface".
      • To fix issue for "Vlan Host Switch interfaces"(fp-ethx) and "Management interface",
      • From VC UI, Disconnect and Re-connect DVPGs connected to "Vlan Host Switch interfaces"(fp-ethx) interfaces.
      • From VC UI, Disconnect and Re-connect DVPGs connected to "Management interface".

After all above 3 Steps are performed then issue should be resolved. "com.vmware.port.extraConfig.vdl2.nestedTNConfig" property with TEP VLAN specified in "Edge Overlay HostSwitch"(e.g. com.vmware.port.extraConfig.vdl2.nestedTNConfig = version=1;vlan=<Edge-TEP-VLAN>,label=28674 , propType = CONFIG) will be visible on DVPG port connected to VTEP interface, Management interface and Vlan Host Switch interfaces. "net-dvs -l command" can be used to verify these DVPG ports. This command need to be run on ESXi Host node on which edge resides.

 

Workaround Steps(When setup is in pinned state and/OR not yet upgraded  to 9.1.0.0 version)

This workaround is to be applied when setup is in pinned state e.g. Setup is upgrading from 4.x to 9.1.0.0 and finalize upgrade step is not yet done.

Step 1 : Change uplink profile to add the transport VLAN

 As shown below in highlighted part, enter non zero VLAN id which you want to use for VTEP interfaces in "Uplink Profile".

Note : This transport VLAN will get updated on all edge transport nodes consuming this uplink profile

If different edge nodes require different transport VLANs, create a separate uplink profile.

 

 

Step 2 : Attach the Edge VTEP vNIC to "VLAN Trunk  DVPG" which contains the VTEP VLAN in the Trunk range

DVPG Configuration before Workaround will look something like below → 

 

  • If DVPG connected to edge VTEP interfaces is a non-trunk as mentioned above. Then we need to make this DVPG as "VLAN Trunk-DVPG" with below steps. 

----------------------------------------------------------------------------------------------------------------

Make DVPG as "VLAN Trunk-DVPG" → 

  • Login to vCenter → Go to Networks → Select DVPG → Then configure VLAN trunk range for this DVPG. In VLAN trunk range just specify edge VTEP VLAN ID as mentioned below 

Connect this "VLAN Trunk-DVPG" to Edge VTEP Interface → 

  • After above configuration we need to attach this VLAN Trunk DVPG to edge VTEP interface i.e. "Uplink interfaces" from Overlay HostSwitch

  • Also, we need to use "Uplink Profile" for which we defined transport VLAN in Step 1

Note: If managers are still in from Version (4.1.x/4.2.x release), then no need to perform step 3. First 2 steps only are sufficient.

 

Step 3 : After above 2 steps are performed, we need to disconnect and re-connect "Vlan Host Switch interfaces" and "Management interfaces"

  • In 1st two steps issue for VTEP fast path(fp-ethx) interface will be resolved

  • In this 3rd step, we are resolving issue for "Vlan Host Switch interfaces"(fp-ethx) and "Management interface".

  • To fix issue for "Vlan Host Switch interfaces"(fp-ethx) and "Management interface",

    • From VC UI, Disconnect and Re-connect DVPGs connected to "Vlan Host Switch interfaces"(fp-ethx) interfaces.

    • From VC UI, Disconnect and Re-connect DVPGs connected to "Management interface".

 

After all above 3 Steps are performed then issue should be resolved. "com.vmware.port.extraConfig.vdl2.nestedTNConfig" property with TEP VLAN specified in "Edge Overlay HostSwitch"(e.g. com.vmware.port.extraConfig.vdl2.nestedTNConfig = version=1;vlan=<Edge-TEP-VLAN>,label=28674 , propType = CONFIG) will be visible on DVPG port connected to VTEP interface, Management interface and Vlan Host Switch interfaces. "net-dvs -l command" can be used to verify these DVPG ports. This command need to be run on ESXi Host node on which edge resides.

 

 

 

 

Additional Information

NSX 9.1.1.0 Release Notes

Powered by Wolken Software