Baremetal Server network behavior when TEP is required to route traffic.

Products

• VMware NSX

Issue/Introduction

When Baremetal Server is configured with VIF. The VIF internet/other network traffic doesn't work consistently. This happens when the TEP default gateway is configured and TEP is required to route the traffic to other TEP in different subnet as well.

For Example consider the following data points: 

• Interface on Baremetal server  "VIF, TEP and Mgmt".
• The routing table looks as follows for Windows and Linux Respectively.

192.168.106.2 is the "VIF on WindowsBM"
192.168.12.6 is the "TEP on WindowsBM"
192.168.200.19 is the "Mgmt interface on WindowsBM"



As per the routing table, the metric for VIF and TEP is 26. This leads to traffic sometimes sourced from VIF and sometimes from TEP.
If we reduce the VIF Metric, the VIF traffic is inconsistent. Meaning, While pinging the IP 8.8.8.8 ; few ping go through and few don't.

Environment

VMware NSX-T Data Center

Cause

It's an expected behavior when the above Symptoms are observed.

Resolution

Add Default route for the TEP traffic while you want both VIF and TEP to communicate with default gateway. 

For example :

In Windows Baremetal,
route add 192.168.12.0 mask 255.255.255.0 192.168.12.1

In Linux Baremetal,
ip route add 192.168.12.0/24 via 192.168.12.1