ESXi update/upgrade fails with the error: “Cannot execute upgrade script on host”

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi upgrade attempts fail regardless of the deployment method used (ISO, offline bundle, or vSphere Lifecycle Manager).
  
  
• Remediation fails with a general system error: A general system error occurred: Remediation failed for host ####
  
  
• The log entries below are recorded in the ESXi host's /var/log/esxupdate.log file:
  
  Error getting data for filesystem on '/vmfs/volumes/########-########-####-############': Cannot open volume: /vmfs/volumes/########-########-####-############, skipping.
  
  
• Checking disk space utilization on the host by running the command 'df -h' returns the error: 
  
  VmFileSystem: Slow refresh failed: Cannot open volume: /vmfs/volumes/########-########-####-############
Error when running esxcli, return status was: 1
Errors:
Error getting data for filesystem on '/vmfs/volumes/########-########-####-############': Cannot open volume: /vmfs/volumes/########-########-####-############, skipping.

Environment

VMware vSphere ESXi 8.x

Cause

• ESXi has a duplicate OSData-########-########-####-############ system partition which is inaccessible, preventing the HA VIB from updating.
  
  
• The duplicate/inaccessible OSData-########-########-####-############, shows in red when checked through command 'ls -lh /vmfs/volumes'.
  
  
• Identify the correct OSDATA by matching the output of the below commands:
  
  vmkfstools -P /vmfs/volumes/bootbank1-UUID | grep 'naa'
vmkfstools -P /vmfs/volumes/bootbank2-UUID | grep 'naa'
vmkfstools -P /vmfs/volumes/OSData1-UUID | grep 'naa'
vmkfstools -P /vmfs/volumes/OSData2-UUID | grep 'naa'

Resolution

Option 1: 

1. Take a backup of ESXi host configuration using below commands:
   
   CRITICAL: Do not skip the configuration backup. Unmounting or removing an active system partition (OSData) will cause an immediate ESXi host failure (PSOD or Hang).
   
   vim-cmd hostsvc/firmware/sync_config
vim-cmd hostsvc/firmware/backup_config
   
   
2. Run the following command and note the UUID in red:
    
   ls -lh /vmfs/volumes
   
   
3. Unmount datastore using below command:
   
   esxcli storage filesystem unmount --volume-uuid=(UUID of affected datastore)
   
   
4. Confirm that the duplicate datastore is gone:
   
   ls -lh /vmfs/volumes
   
   
5. Proceed with ESXi update/upgrade.

Option 2:

     Reboot the host and then retry remediation.

Additional Information

When the host is rebooted, the operating system completely clears its memory and rescans all physical storage devices from scratch during the boot process.

Discarding Invalid Entries: Because the duplicate partition doesn't actually exist as a separate valid physical entity, the fresh boot process ignores the previous stale entry and mounts only the single, correct OSData partition.

Removing the Obstacle: When remediation is attempted after the reboot, the upgrade script no longer encounters the conflicting "ghost" partition, allowing the process to access the correct system partitions and complete successfully.