VMware NSX prepared host may be degraded and dropping packets when transport nodes are in different subnets

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Transport nodes are prepared using an IP pool.
• VMware NSX UI shows the transport nodes are degraded and tunnels are down to remote transport nodes.
• The source and destination transport nodes Tunnel Interfaces (TEP's) are on different subnets.
• Packet may be dropped when checking Overlay Segment uplinkTxInvalidStateDrops counter using:

#nsxcli 
>get segments

<list of segments displayed>

>get segment <segment-uuid> datapath-stats module nsxt-vdl2 drops new

• Check the output here over a period of time and notice the uplinkTxInvalidStateDrops is increasing.

Note: Above command is only available in VMware NSX 4.1.x.

• On the ESXi transport node, when we run the following command we find the Gateway is not configured:

#net-vdl2 -l

• The IP pool used for the transport nodes does not have a gateway configured.

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center 2.x
VMware NSX-T Data Center 4.x

Cause

If the Gateway was not specified in the IP pool used to configure the transport nodes and there are transport nodes in different subnet, then this can lead to tunnels not being formed and packets being dropped.
This can occur when the environment is initially setup with all transport nodes in the same subnet and therefore no Gateway is required, then later additional transport nodes are added and they are in a different subnet, with a different IP pool.

Resolution

This is expected behavior if the transport nodes are in different subnets and no gateway is configured in the IP pool.
Improvements have been introduced in VMware NSX 4.1.2 will alert that a gateway has not been added to the IP pool when preparing a transport node using an IP pool which does not have a gateway.

Workaround:
As a workaround, you can temporarily add a Gateway to the ESXi transport node using the vSphere Networking guide.

For a permanent fix, add the gateway address to the IP Address Pool, then re-sync the transport nodes so that the host configuration is updated.
POST API used to resync the transport nodes: 

POST /policy/api/v1/infra/sites/{site-id}/enforcement-points/{enforcementpoint-id}/host-transport-nodes/{host-transport-node-id}?action=resync_host_config

If the gateway address cannot be added to the IP Address Pool because it is in use, then the transport nodes using the IP Address Pool need to be un-prepared first.
Then the IP Address Pool can be edited to include the gateway address
Then prepare the transport nodes again with the updated IP Address Pool.