Edge node connectivity loss due to incorrect vNIC mapping
search cancel

Edge node connectivity loss due to incorrect vNIC mapping

book

Article ID: 451540

calendar_today

Updated On:

Products

VMware NSX VMware Cloud Foundation

Issue/Introduction

  • The vNICs of Edge VMs are not connected to the configured portgroups if fp-eth do not appear in order in the configuration.

For example, if an Edge node is configured as below,

NVDS1:
Uses fp-eth1 and portgroup 2.

NVDS2:
Uses fp-eth0 and portgroup 1.

Edge VM's vNIC mapping is swapped.

fp-eth0 is connected to portgroup 2 and used by NVDS2.
fp-eth1 is connected to portgroup 1 and used by NVDS1.

Furthermore, if any fp-eth is skipped, fp-eth used by NVDS might not be connected to any portgroups.
For example, if an Edge node is configured as below,

NVDS1:
Uses fp-eth2 and portgroup 1.

NVDS2:
Uses fp-eth3 and portgroup 2.

Edge VM's vNIC mapping is below:

fp-eth0 is connected to portgroup 1 and used by no NVDS.
fp-eth1 is connected to portgroup 2 and used by no NVDS.
fp-eth2 is connected to "none" and used by NVDS1.
fp-eth3 is connected to "none" and used by NVDS2.

  • On the other hand, NVDS on the Edge node is correctly configured to use the configured fp-eth, causing mismatch between NVDS and VM's portgroup.
    As a result, such Edge nodes experiences connectivity issues including:
    • Edge node status is down or degraded.
    • All or some TEP tunnels are down.
    • VLAN uplinks and service ports can not reach VLAN target or physical gateway.
  • Upgraded Edges from NSX 4.x might experience connectivity issues after opening [Edge Node Uplink Mapping] and saving even if no configuration was changed.
    If an Edge is configured like below,

NVDS1:
VLAN TZ, fp-eth0, portgroup 1

NVDS2:
Overlay TZ, fp-eth1, porgroup 2

NSX GUI sends an Edge configuration in which NVDS2 first appears and then NVDS1 appears because [Edge Node Uplink Mapping] GUI shows the overlay NVDS on top.

NVDS2:
Overlay TZ, fp-eth1, portgroup 2

NVDS1:
VLAN TZ, fp-eth0, porgroup 1

As a result, Edge VM's vNIC mapping is below:

fp-eth0 is connected to portgroup 2 and used by NVDS1.
fp-eth1 is connected to portgroup 1 and used by NVDS2.

Environment

NSX 9.x

VCF 9.x

Cause

NSX has a problem in processing API input and generates wrong vNIC mapping.

It decides vNIC mapping in order the portgroups appear in the Edge configuration, resulting in wrong vNIC mapping and connectivity issue if fp-eth do not appear in order in the configuration.

Resolution

Currently there is no resolution.

As a workaround, configure Edge node so that fp-ethN appears in order of fp-eth0, fp-eth1, fp-eth2, fp-eth3, and fp-eth4 and no fp-eth is skipped.

In upgrade cases like explained above, API can be a workaround.

  1. Get the current Edge configuration.
    GET /policy/api/v1/infra/sites/default/enforcement-points/default/edge-transport-nodes/<Edge ID>
  2. In the response, the overlay NVDS comes first and the VLAN NVDS comes next.
    Edit the configuration and swap the NVDS order.
    Example:
    From
            "switch_spec": {
              "switches": [
                {
                  "overlay_transport_zone_paths": [
                    "/infra/sites/default/enforcement-points/default/transport-zones/<Overlay TZ>"
                  ],
                  "pnics": [
                    {
                      "datapath_network_id": "<Overlay portgroup>",
                      "device_name": "fp-eth1",
                      "uplink_name": "Uplink-2"
                    }
                  ],
                  "profile_paths": [
                    {
                      "key": "UplinkHostSwitchProfile",
                      "value": "/infra/host-switch-profiles/<UUID>"
                    }
                  ],
                  "switch_name": "NVDS-Overlay",
                  "tunnel_endpoints": [
                    {
                      "ip_assignment_specs": [
                        {
                          "ip_assignment_type": "StaticIpv4Pool",
                          "ip_pool": "/infra/ip-pools/TEP-IP-Pool"
                        }
                      ]
                    }
                  ]
                },
                {
                  "pnics": [
                    {
                      "datapath_network_id": "<VLAN portgroup>",
                      "device_name": "fp-eth0",
                      "uplink_name": "Uplink-1"
                    }
                  ],
                  "profile_paths": [
                    {
                      "key": "UplinkHostSwitchProfile",
                      "value": "/infra/host-switch-profiles/<UUID>"
                    }
                  ],
                  "switch_name": "NVDS-VLAN",
                  "vlan_transport_zone_paths": [
                    "/infra/sites/default/enforcement-points/default/transport-zones/<VLAN TZ>"
                  ]
                }
              ]
            },
    To
            "switch_spec": {
              "switches": [
                {
                  "pnics": [
                    {
                      "datapath_network_id": "<VLAN portgroup>",
                      "device_name": "fp-eth0",
                      "uplink_name": "Uplink-1"
                    }
                  ],
                  "profile_paths": [
                    {
                      "key": "UplinkHostSwitchProfile",
                      "value": "/infra/host-switch-profiles/<UUID>"
                    }
                  ],
                  "switch_name": "NVDS-VLAN",
                  "vlan_transport_zone_paths": [
                    "/infra/sites/default/enforcement-points/default/transport-zones/<VLAN TZ>"
                  ]
                },
                {
                  "overlay_transport_zone_paths": [
                    "/infra/sites/default/enforcement-points/default/transport-zones/<Overlay TZ>"
                  ],
                  "pnics": [
                    {
                      "datapath_network_id": "<Overlay portgroup>",
                      "device_name": "fp-eth1",
                      "uplink_name": "Uplink-2"
                    }
                  ],
                  "profile_paths": [
                    {
                      "key": "UplinkHostSwitchProfile",
                      "value": "/infra/host-switch-profiles/<UUID>"
                    }
                  ],
                  "switch_name": "NVDS-Overlay",
                  "tunnel_endpoints": [
                    {
                      "ip_assignment_specs": [
                        {
                          "ip_assignment_type": "StaticIpv4Pool",
                          "ip_pool": "/infra/ip-pools/TEP-IP-Pool"
                        }
                      ]
                    }
                  ]
                }
              ]
            },
  3. Appy the configuration.
    PATCH /policy/api/v1/infra/sites/default/enforcement-points/default/edge-transport-nodes/<Edge ID>

API can configure NVDS in order of VLAN and Overlay and it makes NSX find portgroups in the expected order.
Do not open [Edge Node Uplink Mapping] and save it until NSX is upgraded to the fixed version.

Additional Information

For details of the API, refer to the API document.
Read an Edge Transport Node under an Enforcement Point
Patch a new LCM Edge Transport Node under Enforcement Point

Powered by Wolken Software