Host VIB update may fail when upgrading from NSX-T 2.3.x to NSX-T 2.4.0 if VMs are storage vMotioned before host upgrade

Products

• VMware NSX

Issue/Introduction

Symptoms:

Host VIB update fails during in-place upgrade with below error in /var/log/upgrade-coordinator/upgrade-coordinator.log 

Unexpected error while upgrading upgrade unit: Install of offline bundle failed on host 6d1d1810-8884-4f4d-83bc-54ec5a86df3a with error : [LiveInstallationError] Error in running ['/etc/init.d/nsx-datapath', 'start', 'upgrade']: Return code: 1 Output: ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored. ioctl failed: No such file or directory ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored. ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored. ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored. start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath", line 1267, in <module> UnloadKernelModules() File "/etc/init.d/nsx-datapath", line 1083, in UnloadKernelModules unloadModule(modName, modName in ['nsxt-vsip']) File "/etc/init.d/nsx-datapath", line 1068, in unloadModule (moduleName, out.decode())) KernelModulesException: Failed to unload module nsxt-switch-security: vmkload_mod: Cannot remove module nsxt-switch-security: Consumed resource count of module is not zero It is not safe to continue. Please reboot the host immediately to discard the unfinished update. Please refer to the log file for more details..

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 2.x

Cause

Issue occurs only if storage vMotion was performed on the host before doing an in-place upgrade from NSX-T 2.3.x to 2.4 and hosts running ESXi-6.5P03 (build 10884925). The switch security module from NSX-T 2.3 does not get removed if storage vMotion was performed just before the host upgrade. The storage vMotion triggers a memory leak causing the switch security module unload to fail.

Resolution

Currently there is no resolution.

Workaround:

Perform maintenance mode upgrade.
Put the host in maintenance mode, reboot the host, followed by NSX host component upgrade..