NSX Install failed on multiple hosts in cluster after upgrade to ESXi 8.0.2.

Products

• VMware NSX

Issue/Introduction

• NSX version has been upgraded to 4.1.1 followed by vCenter Server to 8.2 and ESXi hosts to 8.0 update 2
• After the upgrade, one of the ESXi cluster's host has failed and remaining clusters have upgraded successfully
• Impacted hosts are showing NSX VIB as 4.1.1.0.0-7.0.22224315 instead of 4.1.1.0.0-8.0.22224315
• Management Plane has detected the VIB mismatch and marked Transport Node State as INSTALL_FAILED
• KCP module failed to unload on the dvs host property messages seen in ESXi host logs in /log/esxupdate.log:
  

  esxupdate[2124680]: KernelModulesException: Failed to unload module nsxt-kcp-22224315: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-kcp-22224315 failed: Busy (bad0004)
  esxupdate[2124680]: Cannot remove module nsxt-kcp-22224315: module symbols in use

  $ grep kcp.enable net-dvs_-l.txt
                  com.vmware.nsx.kcp.enable = true , propType = CONFIG
                  com.vmware.nsx.kcp.enable = true , propType = CONFIG

Cause

The issue is caused when one or more of the modules com.vmware.nsx.spf, com.vmware.nsx.vdl2, com.vmware.nsx.kcp are loaded and enabled and preventing the NSX VIB's from being removed.

 

Resolution

This is a known issue affecting VMware NSX

• To resolve the issue, remove the stale entries before re-preparing the host for NSX by following NSX VIBs are not removed from an ESXi host.
• Once the stale entries are removed, you can proceed with re-installing NSX on the affected ESXi host through NSX UI.

Additional Information

Impact/Risks:
NSXA agent is marked as down and VMs cannot vmotion to the affected ESXi.