Faulty TEP alarm and IPv4 BFD tunnels from TEP are down in VMware Cloud Foundation 9.0

Products

• VMware NSX

Issue/Introduction

A "Faulty TEP" alarm is triggered in the NSX configuration, and IPv4 BFD tunnels from the TEP are down. The following symptoms are observed:

• ESXi hosts communicate successfully with each other over their TEP network (for example, 10.4.172.x).

• Edge nodes communicate successfully with each other over their TEP network (for example, 10.4.171.x).

• Communication between the ESXi hosts and the Edge nodes fails.

• Edge nodes are unable to ping their gateway (for example, 10.4.171.1).

• Packet captures at the ESXi uplink (hosting the edge node) show ARP requests from both sides (Edge and the gateway), but packets are dropped before reaching the gateway, resulting in no replies.

• Further investigation reveals that the Edge TEP interface has an incorrect vLAN ID assigned when verified via cli but has right vLAN assigned when validated the uplink profile of the edge node in UI.
  
  Commands to validate the TEP configurations on the edge:
  SSH to edge via admin shell and run below commands:
  
  > get gateways (validate which vrf is the tunnel vrf)
  > vrf <ID> (enter the vrf number collected from step 1)
  > get interfaces (validate the TEP IP and vLAN assignment)

Environment

VMware NSX

Cause

• This issue occurs because the Edge TEP is incorrectly assigned to the wrong VLAN as seen from CLI (for example, VLAN 166) instead of the required VLAN as seen from UI (for example, VLAN 171), even though the TEP IP addresses belong to the correct VLAN subnet. This mismatch causes ARP requests to drop before reaching the gateway.
  
  
• In VMware Cloud Foundation (VCF) versions 9.0 and 9.0.1, the NSX Edge TEP VLAN configuration is fixed at the time of deployment and in this case the Edge TEP vLAN was changed after deployment (by attaching new uplink profile with new vLAN) resulting in Edge retaining the original vLAN for the TEP's even though the uplink profile was changed later to new vLAN.
  
  
• The current architecture does not support post-deployment modification of the TEP VLAN through standard configuration edits.

Resolution

Currently, the only supported method to change the NSX Edge TEP VLAN in VCF 9.0 and 9.0.1 is to redeploy the Edge cluster with the new VLAN requirements.

Please follow the below KB for resolution steps
Unable to Change NSX Edge TEP VLAN Post-Deployment in VCF 9.0.x