ESXi update/upgrade failing with the error: “Cannot execute upgrade script on host”

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi upgrade fails with any method i.e. using ISO file, offline bundle, using VLCM.

Remediation fails with General system error:

"A general system error occurred: Remediation failed for host xxxxxx"

 

Below log entries are registered in esxupdate.log

Error getting data for filesystem on '/vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx': Cannot open volume: /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx, skipping.

 

Running 'df -h' also gives below error,

VmFileSystem: Slow refresh failed: Cannot open volume: /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx
Error when running esxcli, return status was: 1
Errors:
Error getting data for filesystem on '/vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx': Cannot open volume: /vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx, skipping.

Environment

ESXi 7.x

ESXi 8.x

Cause

ESXi has a duplicate OSDATA-xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx system partition which is inaccessible, preventing the HA vib from updating .

 

The duplicate/inaccessible OSDATA-/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx, shows in red when checked through command 'ls -lh /vmfs/volumes'.

You can also identify the correct OSDATA by matching the output of below commands,

vmkfstools -P /vmfs/volumes/bootbank1-UUID | grep 'naa'
vmkfstools -P /vmfs/volumes/bootbank2-UUID | grep 'naa'
vmkfstools -P /vmfs/volumes/OSdata1-UUID | grep 'naa'
vmkfstools -P /vmfs/volumes/OSdata2-UUID | grep 'naa'

 

Resolution

1. Take the backup of ESXi config using below command,
   (Do not skip this step, as removing/unmounting used OSDATA could get ESXi down.)
   • vim-cmd hostsvc/firmware/sync_config
   • vim-cmd hostsvc/firmware/backup_config
2. Run the following and note the UUID in red: 
   • ls -lh /vmfs/volumes
3. Unmount datastore using below command,
   • esxcli storage filesystem unmount --volume-uuid=(UUID of affected datastore)
4. Now confirm that the duplicate datastore is gone: 
   • ls -lh /vmfs/volumes
5. Proceed with ESXi update/upgrade