Overlay Traffic Disruption Occurs when Local and Remote TEPs are on Different Subnets After Upgrading to NSX 4.1.2 and ESXi 8.x

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• BFD tunnel status on the transport node is displayed as "down."
• Overlay L2/L3 traffic experiences outages.
• Local and Remote TEPs are located in different subnets.
• Hosts, running on ESXi 8.x, have been upgraded to NSX 4.1.2 using maintenance mode.
• The 'net-vdl2 -l' command output indicates the gateway IP is absent, as illustrated in the example below:

 

Environment

VMware NSX 4.1.0

Cause

Upon upgrading hosts using maintenance mode to NSX 4.1.2, the underlay gateway IP/MAC for VTEPs vanishes post-upgrade. This disruption leads to L2/L3 overlay traffic failures in customer setups where local and remote TEPs are situated in different underlay subnets. This problem manifests after the host exits maintenance mode during the upgrade.

The issue arises from the following upgrade workflows:

1. When the ESXi version is already at 8.x and NSX is upgraded to 4.1.2.
2. When the ESXi version is 7.x, followed by an upgrade of NSX to 4.1.2, and subsequently, the ESXi version is upgraded to 8.x. The problem only appears post the upgrade of ESXi to 8.x.

Resolution

This problem has been identified as specific to NSX version 4.1.2. The issue has been resolved in the version 4.1.2.1 EP.

Workaround:

For systems already exhibiting this problem and missing gateway information, place the host into maintenance mode and then reboot. This action will restore the absent gateway IP/MAC.

To prevent encountering this issue, execute the NSX host upgrade using the in-place upgrade method.