Management Interface Guide: DLR Control VM

Products

VMware NSX

Issue/Introduction

When deploying NSX Distributed Logical Router (DLR), one of the steps in the UI wizard (and a mandatory field in the matching API call) is the dvPortgroup for attachment of DLR Control VM's Management Interface.

This article provides the important considerations around the use of this interface that are useful to know.

Symptoms:

Environment

VMware NSX for vSphere 6.0.x
VMware NSX for vSphere 6.1.x

Resolution

Management connectivity

DLR Control VM's routing table is directly related to the networks that are either connected to the given DLR, or can be reached through it. This means that for any route present on DLR, there must be a matching next hop, reachable through one of the DLR’s LIFs.

However, there is no matching LIF for DLR Control VM’s Management Interface because it is a construct that is local to the Control VM. Subsequently, any route with next hop pointing to the Management Interface would be invalid.

What are the effects of this?

Consider a scenario, trying to reach an IP address associated with DLR Control VM Management Interface (IP A) from an IP address in a different subnet (IP B).

Assume that the external routing is set up correctly to steer packets toward IP A through dpPortgroup where the Management Interface is connected so that the packets reach successfully. However, the return traffic would have to follow the DLR’s routing table, probably through the default route pointing to one of the DLR’s uplinks, which is an interface that is different from the Management Interface.

In this scenario, the connectivity will not work because both the DLR Control VM and ESGs have strict Reverse Path Forwarding (RPF) check turned on, which cause them to drop any packet received on an interface that is different from the one that would be used to send the return traffic. In other words, for the above to work, the route towards IP A must have Management Interface as next hop for connectivity to work; but that is not possible because there is no corresponding DLR LIF.

What to do about it?

In addition to VM Console connection, DLR Control VM can be reached from network using one of these options:

Through the Management Interface, but only from an IP address on the same subnet
Through the DLR’s Protocol address, set as part of OSPF or BGP configuration, from any IP address that DLR knows how to get back to.

Note: The DLR Control VM cannot establish network connectivity with any IP address that falls into any subnet configured on any of this DLR’s Internal interfaces. This is because the egress interface for these subnets on DLR Control VM points to the pseudo-interface VDR, which is not connected to the data plane.

If option 2 is selected, you can leave the DLR Control VM’s Management Interface without any IP addresses configured. Benefits of this approach is that you can reach your Control VM from remote subnets, and that you do not require the additional configuration described below.

DLR Management Interface and Route Redistribution

Routing protocols on the DLR run on its Control VM. Any route redistribution also occurs there, which include routes for directly connected subnets. For each of the DLR’s LIFs, there is a subnet configured on the DLR Control VM’s special interface, using which the routing engine identifies them.

When an IP address is configured on the DLR’s Management Interface, this IP’s subnet is also seen as directly connected, which is then redistributed and advertised. This may not be desirable.

There is no LIF on DLR that corresponds to Management Interface. Therefore, if DLR is selected by other routers as a valid best path for the Management Interface subnet, it results in traffic blackholing (if DLR has no route back to the source) or traffic loop, DLR trying to send it back.

If you want to configure an IP address on your DLR’s Control VM and you are redistributing connected routes, you can create an import filter as shown below, where the Management Interface is configured with the IP address of 10.x.x.x/24:

DLR with HA enabled

When DLR is deployed with HA enabled, it will use its Management Interface to exchange HA heartbeat messages to control Active / Standby state of the HA pair of DLR Control VMs. If the communication is broken between the Active and Standby VM, both of them assume Active role, resulting in a split-brain scenario.

Split-brain scenario must be avoided when a DLR is configured for bridging VLANs with VXLANs. If the HA heartbeat is disrupted in this configuration, the two ESXi hosts where DLR Control VMs are running become active bridges between the same set of VLAN+VXLAN, creating a bridging loop(s).

When selecting dvPortgroup for your Management Interface, ensure that it is a part of the same DVS that carries your bridged VLANs and VXLANs, so that in case of physical NIC failure both HA and dataplane connectivity for your bridged VLANs and VXLANs gets cut together, preventing the forwarding loop.