NSX Host Upgrade Fails with the error "Upgrade eligibility not met. Multiple Instances of Datapath Modules present on the host"

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention
• VMware NSX

Issue/Introduction

NSX transport node upgrades from 3.2.x to 4.2.x are failing on several ESXi hosts, with the error: "Upgrade eligibility not met. Multiple instances of datapath modules present on host. Please reboot the host to continue." This issue prevents the upgrade from proceeding.

 

Environment

VMware NSX:
Source version: 3.2.x
Target version: 4.2.x
Components: NSX Manager, NSX Transport Nodes (ESXi hosts), Distributed Firewall (DFW).
Impacted Functionality: NSX Host Upgrade process.

Cause

The root cause is a memory leak within the vsip-fqdn heap on NSX 3.2.x hosts. This occurs under specific conditions:

Active FQDN-based DFW rules: Distributed Firewall rules are configured with L7 context profiles having FQDNs (e.g., for Office 365 domains).
VM vMotion between NSX-T versions: Virtual machines are vMotioned back n forth between the already upgraded 4.2.x ESXi hosts and the to be upgraded 3.2.x ESXi hosts
Continuous DNS activity: Active traffic involving DNS resolution for FQDNs configured in context profiles used in DFW rules, especially CNAME records

During an upgrade from NSX 3.2.x to 4.2.x, if these conditions are met, the nsxt-vsip module (part of the older NSX 3.2.x version) fails to properly release its consumed resources. This prevents the module from being completely unloaded from the ESXi kernel. As a result, when the upgrade attempts to install the new 4.2.x modules, the system detects both old and new modules  simultaneously loaded, triggering the "Multiple instances of datapath modules present on host" error and halting the upgrade. 

Resolution

The current resolution involves rebooting the affected ESXi host:

1. Place the affected ESXi host into Maintenance Mode
2. Reboot the ESXi host
3. Wait for the host to fully boot and reconnect to vCenter Server
4. Retry the NSX-T transport node upgrade for the host
5. This action has been confirmed to allow the upgrade to proceed successfully

Additional Information

Log Evidence for Module Unload Failure:

/var/run/log/vmkernel.log 

2025-09-19T21:19:57.035Z cpu60:98855903)ALERT: Heap: 2793: Non-empty heap (vsip-fqdn) being destroyed (avail is 8389664, should be 8392176).
2025-09-19T21:19:57.035Z cpu60:98855903)UMA Zone 24 (pffqdndomainent), ObjInUse = 6, maxItems = 2147483647, itemSize = 336, failures = 0
2025-09-19T21:19:57.035Z cpu60:98855903)UMA Zone 25 (pffqdnuuidpl), ObjInUse = 6, maxItems = 2147483647, itemSize = 48, failures = 0
2025-09-19T21:19:57.091Z cpu60:98855903)ALERT: Mod: 5062: Failed to unload module nsxt-vsip-22339670, since its consumed resource count is 1. Waiting...
2025-09-19T21:20:02.109Z cpu60:98855903)ALERT: Mod: 5091: Failed to unload module nsxt-vsip-22339670, since its consumed resource count is 1. Giving up
2025-09-19T21:20:02.636Z cpu69:98856074)WARNING: Mod: 4974: Unload of module nsxt-vsip-22339670 failed : Consumed resource count of module is not zero

/var/run/log/syslog <<< Indicates 2 instances of various nsx modules

2025-09-23T02:15:39Z init.d/nsx-esx-precheck: WARNING: 2 instances of nsxt-core module detected
2025-09-23T02:15:39Z init.d/nsx-esx-precheck: WARNING: 2 instances of nsxt-fc module detected
2025-09-23T02:15:39Z init.d/nsx-esx-precheck: WARNING: 2 instances of nsxt-livetrace module detected
2025-09-23T02:15:39Z init.d/nsx-esx-precheck: WARNING: 2 instances of nsxt-traceflow module detected
2025-09-23T02:15:39Z init.d/nsx-esx-precheck: WARNING: 2 instances of nsxt-vswitch module detected