vSphere HA configuration fails with the error Error: Cannot install the vCenter Server agent service. "Unknown installer error"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

• Following error will be observed at the host task list while configuring vSphereHA on the cluster

  Cannot install the vCenter Server agent service . "Unknown installer error"

• FDM vib can be found in the /tmp folder

• Duplicate OSDATA with a broken link and 2 additional bootbank's to be noticed.

• df -h gives the below error

[root@XXXESXi:/vmfs/volumes/671####-######-#####-68####c2bc/log] df -h
VmFileSystem: Slow refresh failed: Cannot open volume: /vmfs/volumes/671####-######-#####-68####c2bc
Error when running esxcli, return status was: 1
Errors:
Error getting data for filesystem on '/vmfs/volumes/671####-######-#####-68####c2bc': Cannot open volume: /vmfs/volumes/671####-######-#####-68####c2bc, skipping.
Cannot open volume:

Environment

• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0
• VMware vSphere vCenter 7.0
• VMware vSphere vCenter 8.0

Cause

• The issue is seen when ESXi has duplicate OSDATA-#####-#####-######-########## system partition and shows inaccessible. This would be preventing the HA vib installation.

• This can be validated from the log /var/run/log/esxupdate.log on ESXI host experiencing the VIB installation/ HA configuration error.

Example log snippet as below.   

[Timestamp] esxupdate: #####: Transaction: INFO: Final list of VIBs being installed: VMware_bootbank_vmware-fdm_#.##.#-#.########
[Timestamp] esxupdate: #####: imageprofile: INFO: Adding VIB VMware_bootbank_vmware-fdm_#.##.#-############# to ImageProfile (Updated) LVO_#.##.#-LVO.###.##.#
[Timestamp] esxupdate: #####: imageprofile: DEBUG: VIBs {'VMware_bootbank_vmware-fdm_#.##.#-########'} are replaced by VIB VMware_bootbank_vmware-fdm_#.##.#-#.########, removing them from ImageProfile (Updated) LVO_#.##.#-LVO.###.##.#
[Timestamp] esxupdate: #####: imageprofile: DEBUG: VIB VMware_bootbank_vmware-fdm_#.##.#-######## is being removed from ImageProfile (Updated) LVO_#.##.#-LVO.###.##.#
[Timestamp] esxupdate: #####: esxupdate: ERROR: An esxupdate error exception was caught:
[Timestamp] esxupdate: #####: esxupdate: ERROR: Traceback (most recent call last):
[Timestamp] esxupdate: #####: esxupdate: ERROR: File "/xxxxx/vmware/esximage/Utils/HostInfo.py", line ###
[Timestamp] esxupdate: #####: esxupdate: ERROR: return runCli(['storage', 'filesystem', 'list'], True)
[Timestamp] esxupdate: #####: esxupdate: ERROR: File "/lib64/pytho0000/ccccx##-s/esx###00.py", line ###
[Timestamp] esxupdate: #####: esxupdate: ERROR: esxutils.EsxcliError: Errors:
[Timestamp] esxupdate: #####: esxupdate: ERROR: Error getting data for filesystem on '/vmfs/volumes/############################': Cannot open volume: /vmfs/volumes/############################, skipping.
[Timestamp] esxupdate: #####: esxupdate: ERROR: Cannot open volume:

• The duplicate/inaccessible  OSDATA-#####-#####-######-##########, shows in red when checked through command 'ls -ltrah'

• You can also identify the correct OSDATA by matching the output of below commands,

vmkfstools -P /vmfs/volumes/bootbank1-UUID
vmkfstools -P /vmfs/volumes/bootbank2-UUID
vmkfstools -P /vmfs/volumes/OSdata1-UUID
vmkfstools -P /vmfs/volumes/OSdata2-UUID

Resolution

1. Take the backup of ESXi config using below command,

   (Do not skip this step, as removing/unmounting used OSDATA could get ESXi down.)

   • vim-cmd hostsvc/firmware/sync_config

   • vim-cmd hostsvc/firmware/backup_config

2. Run the following and note the UUID in red: 

   • ls -lh /vmfs/volumes

3. Unmount datastore using below command,

   • esxcli storage filesystem unmount --volume-uuid=(UUID of affected datastore)

4. Now confirm that the duplicate datastore is gone: 

   • ls -lh /vmfs/volumes

5. Reboot the ESXi host to remove the duplicate bootbank's listed on the ESXi host.

Additional Information

Reference: VmFileSystem: Slow refresh failed