NSX BFD tunnels may go down when upgrading from version 4.2.1.1 to 9.0.2

Products

• VMware NSX

Issue/Introduction

• Some TEP Geneve overlay tunnels on one of the VTEP interfaces, for example vmk11, will show a status of 'Down' in the NSX Manager UI due to BFD sessions being 'Down'.
• Due to this affected hosts are reported as being in a 'Degraded' state.
• The issue affects a significant number of hosts across multiple clusters.
• Issue is observed post NSX Manager and Edge upgrade but prior to Host Transport Node upgrade, so the Hosts are still on the 4.2.1.1 vibs.
• Rebooting all the NSX Managers in the cluster does not resolve the issue.
• Vmkping tests between the affected VTEP interfaces are successful. The relevant vmkping test steps are described in the KB 379112.
• In the affected hosts /var/run/log/vmkernel logs, a teaming policy update applied globally to the logical switch is logged a few seconds prior to the BFD tunnel being reported as DOWN:

In(182) vmkernel: cpu50:2103184)Team.vswitch: TeamVS_LSPolicyUpdate:10751: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Updating logical switch teaming policy, ps DvsPortset-X
In(182) vmkernel: cpu50:2103184)Team.vswitch: TeamVS_LSPolicyUpdate:10867: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Modify the LS teaming policy Uplink-X on DvsPortset-X
In(182) vmkernel: cpu50:2103184)Team.vswitch: TeamVS_LSPolicyUpdate:10867: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Modify the LS teaming policy Uplink-X on DvsPortset-X
In(182) vmkernel: cpu50:2103184)Team.vswitch: TeamVS_LSPolicyUpdate:10867: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Modify the LS teaming policy Uplink-X-<POLICY_NAME> on DvsPortset-X
In(182) vmkernel: cpu50:2103184)Team.vswitch: TeamVS_LSPolicyUpdate:10867: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Modify the LS teaming policy Uplink-X-<POLICY_NAME> on DvsPortset-X
In(182) vmkernel: cpu50:2103184)Team.vswitch: TeamVS_LSPolicyUpdate:10898: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Successfully updated logical switch teaming policies, ps DvsPortset-X

Environment

VMware NSX

Cause

• This issue is under active investigation by Engineering. Subscribe to this article to be updated on the final fix status. See How to subscribe to a Knowledge Management (KM) article.
• Collect the following data for Support and open a new case, if experiencing this issue, before applying the workaround in the resolution section:
  • Simultaneous pktcap-uw captures on the affected VTEP vmk interface and the physical pNIC uplinks assigned to the Virtual Distributed Switch:
    1. On the local ESX host, capture the BFD packets using the pktcap-uw tool from the ESX command line:
       - pktcap-uw --vmk vmk<10 or 11> --dir 2 --ip remote-vtep-ip -o /vmfs/volumes/<DATASTORE_NAME>/<HOSTNAME_vmk>.pcap & pktcap-uw --uplink <VMNIC(X)> --dir 2 --ip remote-vtep-ip -o /vmfs/volumes/<DATASTORE_NAME>/<HOSTNAME_vmnic(X)>.pcap & pktcap-uw --uplink <VMNIC(X)> --dir 2 --ip remote-vtep-ip -o /vmfs/volumes/<DATASTORE_NAME>/<HOSTNAME_vmnic(X)>.pcap &
    2. On the peer ESX host, capture the same BFD packets:
       - pktcap-uw --vmk vmk<10 or 11> --dir 2 --ip remote-vtep-ip -o /vmfs/volumes/<DATASTORE_NAME>/<HOSTNAME_vmk>.pcap & pktcap-uw --uplink <VMNIC(X)> --dir 2 --ip remote-vtep-ip -o /vmfs/volumes/<DATASTORE_NAME>/<HOSTNAME_vmnic(X)>.pcap & pktcap-uw --uplink <VMNIC(X)> --dir 2 --ip remote-vtep-ip -o /vmfs/volumes/<DATASTORE_NAME>/<HOSTNAME_vmnic(X)>.pcap &
  • A live coredump from the affected hosts:
    1. Run the following command to dump the memory (Note this action will not interrupt the VMs):
       - localcli --plugin-dir /usr/lib/vmware/esxcli/int debug livedump perform
    2. Run the following command to copy the memory dump to the active dump partition:
       - esxcfg-dumppart -C -D active
    3. Export the host logs using the vCenter UI, this will include the 'CoreDumps' by default.
    4. Export NSX Manager logs.
• To speak to a Technical Support Engineer if there are any issues or questions related to using the above commands, see KB 206567 - Contact Broadcom support.

Resolution

There is no fix available for this issue. Use on of the following workarounds to continue with the upgrade.

Workaround 1: Restart NSX Services:

1. Log in to the affected ESXi host via SSH as root.
2. Restart the following NSX services:
   • /etc/init.d/nsx-opsagent restart
   • /etc/init.d/nsx-proxy restart
   • /etc/init.d/nsx-cfgagent restart
   • /etc/init.d/nsx-nestdb restart
3. Verify the services are running:
   • /etc/init.d/nsx-opsagent status
   • /etc/init.d/nsx-proxy status
   • /etc/init.d/nsx-cfgagent status
   • /etc/init.d/nsx-nestdb status

Workaround 2: Maintenance Mode Toggle:

1. In the vSphere Client, right-click the affected host and select Maintenance Mode > Enter Maintenance Mode.
2. Once the host has entered Maintenance Mode, right-click and select Maintenance Mode > Exit Maintenance Mode.

Additional Information

• Details on how to use the pktcap-uw tool, see KB 341568.
• How to generate live core dump on ESX host, see KB 340041.