NSX Install failed on multiple hosts in cluster after upgrade to ESXi 8.0.2.

Products

• VMware NSX

Issue/Introduction

• NSX version has been upgraded to 4.1.1 followed by vCenter Server to 8.2 and ESXi hosts to 8.0 update 2
• After the upgrade, one of the ESXi cluster's host has failed and remaining clusters have upgraded successfully
• Impacted hosts are showing NSX VIB as 4.1.1.0.0-7.0.22224315 instead of 4.1.1.0.0-8.0.22224315
• Management Plane has detected the VIB mismatch and marked Transport Node State as INSTALL_FAILED

Environment

VMware NSX

Cause

KCP module failed to unload on the dvs host property:

esxupdate[2124680]: KernelModulesException: Failed to unload module nsxt-kcp-22224315: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-kcp-22224315 failed: Busy (bad0004)
sxupdate[2124680]: Cannot remove module nsxt-kcp-22224315: module symbols in use

$ grep kcp.enable net-dvs_-l.txt
                com.vmware.nsx.kcp.enable = true , propType = CONFIG
                com.vmware.nsx.kcp.enable = true , propType = CONFIG

Resolution

Workaround:

1. Follow KB to unload the kcp module from the vSwitch: Uninstalling NSX-T from ESXi host failing with error "Failed to remove all host switches or logical switches" 
2. Remove NSX from host using "del nsx"
3. Reboot the ESXi host
4. Install NSX on the host via NSX User Interface

Additional Information

Impact/Risks:
NSXA agent is marked as down and VMs cannot vmotion to the affected ESXi.