NSX-T dataplane outage during management plane upgrade

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• During upgrade from NSX-T 2.4.x to 2.5.x management plane, you may experience dataplane outage.
• In the ESXi nsx-syslog.log log file, you can see the vdr mac address being cleared:
  • <182>1 2021-03-18T19:15:39.514Z esxi-01.corp.local NSX 2104344 - [nsx@6876 comp="nsx-esx" subcomp="opsagent" s2comp="nsxa" tid="2104601" level="INFO"] Dvs [75 80 fb de d8 92 46 34-a4 ac 26 a0 00 59 96 b1] VDR MAC will change from [02:50:56:56:44:52] to [].
• Above we can see the command set the ESXi VDR mac address to null ([]) from [02:50:56:56:44:52].

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center

Cause

The management plane sent down an incorrect message to clear the vdr mac address on the ESXi hosts.
At this point, the management plane upgrade was in progress and controllers were down.
That meant the ESXi hosts could not get the correct mac address again until the management plane upgrade was complete and controllers were back up.
This issue only occurs in upgrades from 2.4.x to 2.5.x only.

Resolution

This issue only occurs in NSX-T management plane upgrade from versions 2.4.x to 2.5.x, it is a timing issue and may not be encountered every time.
For 2.5.x code was introduced which means if the vdr mac address is null, it will resolve this by setting the default mac address.

Workaround:
Currently there is no workaround.
When management plane upgrade is complete and controllers are back on line, the ESXi hosts will get the correct vdr mac address.
It is advised to upgrade in a maintenance window to avoid dataplane outage.