BFD tunnels between Edge nodes on same cluster remained down after a power outage

Products

• VMware NSX
• VMware Telco Cloud Automation

Issue/Introduction

After recovering from a power outage, some of the control plane nodes and worker nodes were not reachable with each other which resulted in TCA cluster instability. 

Issue from TCA is external IP was not getting assigned to the newly created node 

The tier-1 and tier-0 gateways for some of the segments which assigns DHCP IPs to control plane nodes and worker nodes are deployed on these edge nodes.

Environment

NSX 4.2.1.4

TCA 2.3

Cause

The control plane and worker nodes receive DHCP IP address from the DHCP server configured on the segment which are deployed on the tier1 routers. 

These tier1 routers are deployed on the Edge nodes where BFD/TEP tunnels remained down after power outage.

Resolution

Migrate/vmotion NSX edge nodes to different ESXi hosts and ensure the BFD/TEP tunnels comes up between NSX edge nodes and also with ESXi hosts 

Make sure the DHCP IP address is assigned to the control plane and worker nodes and they are able to reach each other and the cluster becomes stable.

Additional Information

Refer KB KB 419021 for more information