Understanding Segment distribution on Edges for RTEP traffic in an NSX Federation envirment

Products

• VMware NSX

Issue/Introduction

• NSX Federation
• Cross site stretched Segments
• Each stretched Segment cross-location communication is distributed across different Edge Nodes in Active/Standby mode to offer Edge Node load distribution.
• After a HA Edge failover the segments are no longer distributed across edge nodes and all segments maybe observed Active in the same Edge.

 

Example:

   >get vtep-group

   VNI           IP                          MAC                             LABEL                      Segment                        TransportNode-Id                                                TN-Connection   HA-STATE
  12###5   1##.###.#40.#05     00:50:56:XX:XX:XX      0x1###5                  1##.###.#40.0                 ########-####-####-####-########b570     true                     ACTIVE
  12###5   1##.###.#40.106     00:50:56:XX:XX:XX      0x1###6                  1##.###.#40.0                 ########-####-####-####-########4905       false                   STANDBY

   VNI           IP                          MAC                             LABEL                      Segment                        TransportNode-Id                                                TN-Connection   HA-STATE
  3###1    1##.###.#40.#05     00:50:56:XX:XX:XX       0x1###5                  1##.###.#40.0                 ########-####-####-####-########b570       true                      ACTIVE
  3###1    1##.###.#40.#04     00:50:56:XX:XX:XX       0x1###4                  1##.###.#40.0                 ########-####-####-####-########4905       false                    STANDBY

   VNI           IP                          MAC                             LABEL                      Segment                        TransportNode-Id                                                TN-Connection   HA-STATE
  6###7    1##.###.#40.#05     00:50:56:XX:XX:XX       0x1###5                  1##.###.#40.0                 ########-####-####-####-########b570       true                      ACTIVE
  6###7    1##.###.#40.#06     00:50:56:XX:XX:XX       0x1###6                  1##.###.#40.0                 ########-####-####-####-########4905       false                    STANDBY

Environment

• VMware NSX-T 3.X
• VMware NSX 4.X

Cause

The Edge Node active/standby pair in charge of a cross-location Stretched-Segment does not support preemptive mode.
In case of an Edge Node failure or reboot (ESXi failure, Edge upgrade, etc), all the cross-location Stretched-Segments it was in charge of are transferred to the other Edge Node pair.
But even after the Edge recovery, no cross-location Stretched-Segments will go back to that Edge.
So after an Edge failure or reboot, there is no longer a fair distribution across Edges of the cross-location Stretched-Segments.

Resolution

This behavior is by design.

Additional Information

How to get a list of the Edge Node load distribution for all segments

On Local Manager Commnad line, with root user:

# su admin -c get vtep-group | awk '{print $1}' | grep -v VTEP-GROUP-ID > /image/VTEP-GROUP-ID
# for i in $(cat /image/VTEP-GROUP-ID); do su admin -c get vtep-group $i vtep ;done