Adding ESXi host with a large number of datastores to NSX prepared cluster fails

Products

• VMware NSX

Issue/Introduction

• In the NSX Manager UI, the NSX Configuration status shows Install Failed.
• The ESXi host  is configured with a large number (400+) of mounted NFS or VMFS datastores.
• nsxapi log indicates NSX VIB installation failed.
  /var/log/proton/nsxapi.log
<TIMESTAMP> INFO workerTaskExecutor-#-# DeploymentUnitInstanceWorker 5306 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] hostId <TN_ID> [fabricUp false, fabricDown false]
duInstance DeploymentUnitInstance [ id=DeploymentUnitInstance/<DUI_ID>, deploymentUnitId=DeploymentUnit/<DU_ID>, hostId=HostTransportNode/<TN_ID>, entityId=null, prevEntityId=null, runningVersion=null, deploymentProgressState=INSTALL_FAILED, deploymentGoalState=ENABLED, internalLastKnownOSVersion=8.0.2, agentId=null, errorId=26050, errorMessage=Operation timed out.]
  
  
• hostd shows it took approximately 9 minutes to complete NSX VIB Installation.
  /var/log/hostd.log
<DATE>T00:28:57.593Z In(166) Hostd[<PID>]: [Originator@6876 sub=Solo.VmwareCLI opID=<OP_ID> sid=<SID> user=nsxuser] Dispatch software.component.apply
...
<DATE>T00:37:51.028Z In(166) Hostd[<PID>]: [Originator@6876 sub=Solo.VmwareCLI opID=<OP_ID> sid=<SID> user=nsxuser] Dispatch software.component.apply done
  
  
• syslog shows continuous SlowRefresh while software.component.apply:
  /var/log/syslog
<DATE>T00:29:02.051Z In(14) ConfigStore[<PID>]: GetTypedFileSystems: fstype PMEM
<DATE>T00:29:02.438Z In(14) ConfigStore[<PID>]: SlowRefresh: path /vmfs/volumes/<VOLUME> total blocks <TOTAL_BLOCKS> used blocks <USERD_BLOCKS>forceRefresh = 0
<DATE>T00:29:02.641Z In(14) ConfigStore[<PID>]: SlowRefresh: path /vmfs/volumes/<VOLUME> total blocks <TOTAL_BLOCKS> used blocks <USERD_BLOCKS>forceRefresh = 0
<DATE>T00:29:02.844Z In(14) ConfigStore[<PID>]: SlowRefresh: path /vmfs/volumes/<VOLUME> total blocks <TOTAL_BLOCKS> used blocks <USERD_BLOCKS>forceRefresh = 0
<DATE>T00:29:03.232Z In(14) ConfigStore[<PID>]: SlowRefresh: path /vmfs/volumes/<VOLUME> total blocks <TOTAL_BLOCKS> used blocks <USERD_BLOCKS>forceRefresh = 0
...
<DATE>T00:34:25.447Z In(14) ConfigStore[<PID>]: SlowRefresh: path /vmfs/volumes/<VOLUME> total blocks <TOTAL_BLOCKS> used blocks <USERD_BLOCKS>forceRefresh = 0
<DATE>T00:34:28.110Z In(14) ConfigStore[<PID>]: SlowRefresh: path /vmfs/volumes/<VOLUME> total blocks <TOTAL_BLOCKS> used blocks <USERD_BLOCKS>forceRefresh = 0
<DATE>T00:34:30.840Z In(14) ConfigStore[<PID>]: SlowRefresh: path /vmfs/volumes/<VOLUME> total blocks <TOTAL_BLOCKS> used blocks <USERD_BLOCKS>forceRefresh = 0
<DATE>T00:34:33.476Z In(14) ConfigStore[<PID>]: SlowRefresh: path /vmfs/volumes/<VOLUME> total blocks <TOTAL_BLOCKS> used blocks 3<USERD_BLOCKS>forceRefresh = 0

Environment

NSX 4.1
ESXi 8.0

Cause

The issue occurs because the ESXi host takes longer than the default 10-minute (600 seconds) timeout period to complete the installation of NSX VIBs.

Analysis of nsxapi.log and hostd.log confirms that while the software.component.apply task on the ESXi host eventually finishes successfully, the total time elapsed exceeds the NSX Manager's threshold. Consequently, the Manager cancels the installation phase and reports a timeout before the host can send a successful heartbeat.

Resolution

Since the VIB installation usually completes successfully on the ESXi host despite the timeout reported by the Manager, the registration can be finalized by manually triggering the "Resolve" action.

1. Log in to the NSX Manager UI.
2. Navigate to System > Fabric >Hosts > Clusters
3. Locate the affected host showing Install Failed.
4. Click on the Install Failed status or the warning icon to open the error details.
5. Click the Resolve button.

The NSX Manager will re-verify the status of the host. Since the VIBs were already applied during the first attempt, the second synchronization will complete within the timeout period, and the status will transition to Success / Up.

Alternatively, reduce a number of datastores mounted on the ESXi host.