ESXi host lost management connectivity during maintenance mode VMware NSX upgrade

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

• Upgrading NSX to version 4.x.
• You are doing a upgrade using maintenance mode.
• The host management vmkernel are using a LACP (LAG).
• During the upgrade, the host goes into maintenance mode and then losses network connectivity.
• If the management vmkernel does not use lacp and other vmkernels do, ESXi management connectivity may continue to work, but other services such as vMotion or storage may be impacted.
• The NSX-T upgrade UI may display the following log entry:
  • Unexpected error while upgrading upgrade unit: Install of offline bundle failed on host <host-uuid> with error : VI SDK invoke exception:java.rmi.RemoteException: VI SDK invoke exception:org.dom4j.DocumentException. Please refer Recovering from an NSX-T In-place Upgrade Failure article for troubleshooting steps.

Note: The KB in the log entry is unrelated, as this is not an in place upgrade.

• In the NSX-T manager /var/log/syslog contains alerts such as:

NSX 73274 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="ccp"] Connection closed received NettyConnection(NettyChannel(local=<ManagerNodeIP>:1235, remote=<TransportNodeIP>:45998), active=false)

• In the ESXi log /var/run/log/vmkernel.log contains entries similar to:

cpu33:223985481)Team.vswitch: TeamVSLACPLAGEventCB:9087: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Received event LAG DESTROY, LAG /0, link UNKNOWN, uplink /0x0, link UNKNOWN
cpu34:223985789)kcp: KCPSHARegisterEvent:545: [nsx@6876 comp="nsx-esx" subcomp="kcp"]KCP_SHA register VMK_PORTSET_EVENT_LACP_LAG event success
cpu6:223985481)Net: 2184: connected LACP_MgmtPort  to null config, portID 0x4000032
cpu6:223985481)Team.vswitch: TeamVSLACPLAGEventCB:9119: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Received event LAG CREATE, LAG /0, link UNKNOWN, uplink /0x0, link UNKNOWN
...
cpu6:223985481)Team.vswitch: TeamVSPolicySet:8123: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Invalid Uplink : <LAG-NAME>, ignore it
cpu6:223985481)Team.vswitch: TeamVSPolicySet:8123: [nsx@6876 comp="nsx-esx" subcomp="vswitch"]Invalid Uplink : <LAG-NAME>, ignore it

Environment

VMware NSX 4.x
VMware NSX-T Data Center

Cause

Due to a race condition, when using a lag, the lag may break during the NSX upgrade.
This can occur under the following conditions:

• A Maintenance Mode upgrade of the ESXi using a lag is carried out, which is the default upgrade.
• VDS/CVDS is used and the vmkNIC is on an ESXi owned DVPG (as opposed to an NSX segment)
• LAG/LACP is configured as an uplink for the DVPG

Resolution

This issue is resolved in VMware NSX 4.1.2.4
This issue is resolved in VMware NSX 4.2.0

Workaround:

To prevent this issue from occurring, set the ESXi host to upgrade in "In-place" mode. In this mode, the ESXi host will not enter maintenance mode and will not trigger the script which leads to the race condition.

If the host is already upgraded when this is issue occurs, reboot the ESXi host to regain management connectivity.

Please note that In-place mode is not supported mode of upgrade for VMware Lifecycle Manager (vLCM) enabled clusters.