ESXi host lost management connectivity during maintenance mode VMware NSX upgrade

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

• Upgrading NSX to version 4.x.
• You are doing a upgrade using maintenance mode.
• The host management vmkernel are using a lacp (lag).
• During the upgrade, the host goes into maintenance mode and then losses network connectivity.
• If the management vmkernel does not use lacp and other vmkernels do, ESXi management connectivity may continue to work, but other services such as vMotion or storage may be impacted.
• The NSX-T upgrade UI may display the following log entry:
  • Unexpected error while upgrading upgrade unit: Install of offline bundle failed on host <hsot-uuid> with error : VI SDK invoke exception:java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException. Please refer 'https://kb.vmware.com/s/article/91383' article for troubleshooting steps.

Note: The KB in the log entry is unrelated, as this is not an in place upgrade.

• In the NSX-T manager /var/log/syslog, we see alerts such as:

NSX 73274 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Connection closed received NettyConnection(NettyChannel(local=<ManagerNodeIP>:1235, remote=<TransportNodeIP>:45998), active=false)

• In the ESXi log /var/run/log/vmkernel.log you see entries similar to:

cpu33:223985481)Team.vswitch: TeamVSLACPLAGEventCB:9087: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Received event LAG DESTROY, LAG /0, link UNKNOWN, uplink /0x0, link UNKNOWN
cpu34:223985789)kcp: KCPSHARegisterEvent:545: [nsx@6876 comp="nsx-esx" subcomp="kcp"]KCP_SHA register VMK_PORTSET_EVENT_LACP_LAG event success
cpu6:223985481)Net: 2184: connected LACP_MgmtPort  to null config, portID 0x4000032
cpu6:223985481)Team.vswitch: TeamVSLACPLAGEventCB:9119: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Received event LAG CREATE, LAG /0, link UNKNOWN, uplink /0x0, link UNKNOWN
...
cpu6:223985481)Team.vswitch: TeamVSPolicySet:8123: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Invalid Uplink : <LAG-NAME>, ignore it
cpu6:223985481)Team.vswitch: TeamVSPolicySet:8123: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Invalid Uplink : <LAG-NAME>, ignore it

Environment

VMware NSX-T Data Center 4.x
VMware NSX-T Data Center

Cause

Due to a race condition, when using a lag, the lag may break during the NSX upgrade.
This can occur under the following conditions:

• A Maintenance Mode upgrade of the ESXi using a lag is carried out, which is the default upgrade.
• VDS/CVDS is used and the vmknic is on a ESXi owned DVPG (as opposed to an NSX segment)
• LAG/LACP is configured as an uplink for the DVPG

Resolution

This issue is resolved in VMware NSX 4.1.2.4
This issue is resolved in VMware NSX 4.2

Workaround:
To prevent this issue from occurring, you can set the ESXi host to upgrade in In-place mode, in this mode the host will not enter maintenance mode and will not trigger the script which leads to the race condition.

If you have already upgrade the host and encountered this issue, to regain management connectivity to the ESXi again, you can, Reboot the ESXi host.

Please note that In-place mode is not supported mode of upgrade for VMware Lifecycle Manager (vLCM) enabled clusters.