vSphere HA configuration fails after vCenter upgrade with "Cannot install the vCenter Server agent service"
search cancel

vSphere HA configuration fails after vCenter upgrade with "Cannot install the vCenter Server agent service"

book

Article ID: 389831

calendar_today

Updated On:

Products

VMware vCenter Server 8.0 VMware vCenter Server 7.0 VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0

Issue/Introduction

  • After upgrading the vCenter Server, configuring vSphere High Availability (HA) fails at the cluster level with the following error:

          Error Message:
          "Cannot install the vCenter Server agent service. Unknown installer error System"

  •  Manually installing the vSphere Fault Domain Manager (FDM) agent on the ESXi host fails with the following error:  
    [root@esxi] esxcli software vib install -n vmware-fdm

    [InstallationError]

    Failed to query file system stats: Errors:

    Error getting data for filesystem on '/vmfs/volumes/XXXXXXXX-XXXXXXX-XXXXXX-XXXXXc7b1': Cannot open volume: /vmfs/volumes/XXXXXXXX-XXXXXXX-XXXXXX-XXXXXc7b1, skipping.
  • From /var/run/log/esxupdate.log, the following error can be observed.   
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: imageprofile: DEBUG: VIBs {'VMware_bootbank_vmware-fdm_7.0.3-24201994'} are replaced by VIB VMware_bootbank_vmware-fdm_8.0.3-1.24022515, removing them from ImageProfile (Updated) DEL-ESXi-703_21313628-A11
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: imageprofile: DEBUG: VIB VMware_bootbank_vmware-fdm_7.0.3-24201994 is being removed from ImageProfile (Updated) DEL-ESXi-703_21313628-A11
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR: An esxupdate error exception was caught:
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR: Traceback (most recent call last):
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Utils/HostInfo.py", line 217, in _getFileSystems
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:     return runCli(['storage', 'filesystem', 'list'], True)
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/esxutils.py", line 220, in runCli
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR: esxutils.EsxcliError: Errors:
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR: Error getting data for filesystem on '/vmfs/volumes/XXXXXXXX-XXXXXXX-XXXXXX-XXXXXc7b1': Cannot open volume: /vmfs/volumes/XXXXXXXX-XXXXXXX-XXXXXX-XXXXXc7b1, skipping.
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR: During handling of the above exception, another exception occurred:
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR: Traceback (most recent call last):
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 222, in main
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:     cmd.Run()
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py", line 153, in Run
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 852, in InstallVibsFromSources
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:     inst, removed, exitstate = self._installVibs(curprofile,
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1098, in _installVibs
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:     exitstate = self._validateAndInstallProfile(
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1267, in _validateAndInstallProfile
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:     resVibCache = ReservedVibCache()
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/ImageManager/HostSeeding.py", line 995, in __init__
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:     self._loadCachedVibs()
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/ImageManager/HostSeeding.py", line 1032, in _loadCachedVibs
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:     for fsPath in HostInfo.GetVmfslFileSystems():
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Utils/HostInfo.py", line 260, in GetVmfslFileSystems
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:     return fs['Mount] Point'] for fs in _getFileSystems()
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:   File "/lib64/python3.8/site-packages/vmware/esximage/Utils/HostInfo.py", line 220, in _getFileSystems
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR:     raise Errors.InstallationError(e, None, msg)
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR: vmware.esximage.Errors.InstallationError: Failed to query file system stats: Errors:
    YYYY-MM-DDThh:hh:ssZ esxupdate: 33423039: esxupdate: ERROR: Error getting data for filesystem on '/vmfs/volumes/XXXXXXXX-XXXXXXX-XXXXXX-XXXXXc7b1': Cannot open volume: /vmfs/volumes/XXXXXXXX-XXXXXXX-XXXXXX-XXXXXc7b1, skipping.

     

  • From /var/run/log/vmkernel.log, the following error can be observed

    YYYY-MM-DDThh:hh:ssZ cpu43:11717809)  [HB state abcdef02 offset 3702784 gen 5 stampUS 10286240757290 uuid 67211de9-8dd7a1ea-850a-34800dea3d74 jrnl <FB 0> drv 24.82 lockImpl 4 ip 10.180.5.177]
    YYYY-MM-DDThh:hh:ssZ cpu43:11717809)WARNING: HBX: 2445: Failed to initialize VMFS distributed locking on volume XXXXX-XXXXX-XXXXX-XXX: Read only
    YYYY-MM-DDThh:hh:ssZ cpu43:11717809)Vol3: 4462: Failed to get object 28 type 1 uuid XXXXX-XXXXX-XXXXX-XXX FD 0 gen 0 :Read only
    YYYY-MM-DDThh:hh:ssZ cpu43:11717809)WARNING: Fil3: 1539: Failed to reserve volume f532 28 1 XXXXX XXXXX XXXXX XXXXX0 0 0 0 0 0 0
    YYYY-MM-DDThh:hh:ssZ cpu43:11717809)Vol3: 4462: Failed to get object 28 type 2 uuid XXXXX-XXXXX-XXXXX-XXX FD 4 gen 1 :Read only
    YYYY-MM-DDThh:hh:ssZ cpu0:2097238)ScsiDeviceIO: 4167: Cmd(0x45b99a186a88) 0x89, CmdSN 0x2d from world 11718443 to dev "naa.600xxxxxxxxxxxxxxxxxxxxxxxxxx04" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x7 0x27 0x0
    YYYY-MM-DDThh:hh:ssZ cpu13:2097234)NMP: nmp_ThrottleLogForDevice:3798: last error status from device naa.600xxxxxxxxxxxxxxxxxxxxxxxxxx04 repeated 20 times

Cause

The issue occurs because the LUN presented to the ESXi host is in a Read-Only state, preventing the FDM agent from installing or updating.

Resolution

To resolve the issue:

  • Verify the datastore accessibility and check if it is in a read-only state.

  • Contact your storage vendor to rectify the LUN and ensure it is writable.

  • Once the LUN issue is resolved, retry the vSphere HA configuration.
Powered by Wolken Software