After upgrading to vCenter Server 7.0.0b with vSphere Lifecycle Manager Images configuring the vSphere HA Agent fails

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• After upgrading to vCenter Server 7.0.0b with:
  • Clusters that are using vSphere Lifecycle Manager Images that are configured with vSphere HA
  • HPE Addon version 700.0.0.10.5.0-108
• You see tasks failing similar to:

Cannot complete the configuration of the vSphere HA agent on the host.

Applying HA VIBs on the cluster encountered a failure

Environment

VMware vCenter Server 7.0.x

Cause

This issue is caused by a bad checksum of the Marvell qlnativefc driver in the depot that prevents the Marvell VIB from being downloaded, causing the Apply task to the vSphere Lifecycle Manager clusters timing out.

This can be identified in /storage/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log:

[2020-05-26 12:47:42:215 'HostUpdateDepotManager' 139833857431296 ERROR]  [patchDepotManager, 500] Failed to download the vibs: https://hostupdate.vmware.com/software/VUM/PRODUCTION/addon-main/addon/HPE/vib20/qlnativefc/Marvell_bootbank_qlnativefc_4.1.9.0-1OEM.700.1.0.15525992.vib
[2020-05-26 12:47:42:229 'TaskStatsCollector' 139834339747584 WARN]  [taskStatsCollector, 192] Task type or creation time not present
[2020-05-26 12:47:42:231 'PM.AsyncTask.ClusterApplyHATask{10}' 139833857431296 INFO]  [vciTaskBase, 1343] SerializeToVimFault fault: (vmodl.fault.SystemError) {    faultCause = (vmodl.MethodFault) null,     faultMessage = (vmodl.LocalizableMessage) [       (vmodl.LocalizableMessage) {          key = "com.vmware.vcIntegrity.lifecycle.depots.VibDownloadError",           arg = <unset>,           message = <unset>       }    ],     reason = "vLCM Task failed, see Error Stack for details."    msg = "{     "data": null,     "error_type": "ERROR",     "messages": [         {             "args": [],             "default_message": "Failed to download some vib.",             "id": "com.vmware.vcIntegrity.lifecycle.depots.VibDownloadError"         }     ] }" } Converted fault: (vim.fault.ExtendedFault) {    faultCause = (vmodl.MethodFault) null,     faultMessage = (vmodl.LocalizableMessage) [       (vmodl.LocalizableMessage) {          key = "com.vmware.vcIntegrity.lifecycle.depots.VibDownloadError",           arg = <unset>,           message = <unset>       }    ],     faultTypeId = "SystemError",     data = (vim.KeyValue) [       (vim.KeyValue) {          key = "faultCause",           value = ""       },        (vim.KeyValue) {          key = "reason",           value = "vLCM Task failed, see Error Stack for details."       }    ]    msg = "{     "data": null,     "error_type": "ERROR",     "messages": [         {             "args": [],             "default_message": "Failed to download some vib.",             "id": "com.vmware.vcIntegrity.lifecycle.depots.VibDownloadError"         }     ] }" }

Resolution

To resolve this issue, update the Vendor Add-On for the affected vSphere Lifecycle Manager Image-based clusters.

Workaround:
To workaround this issue, clean the HPE VIBs using the scripts provided:

1. Download and extract the scripts from the attached zip file 79388_remove-HPE-scripts.zip.
2. Copy and execute the remove-HPE.sh script on your vCenter Server Appliance.
3. If UMDS is in use, copy and execute the remove-umds-HPE.sh script on the machine where UMDS is installed to remove obsolete HPE addon from UMDS patch store.
4. After the remove-HPE.sh script automatically restarts vLCM, run Lifecycle Manager > ACTIONS > Updates > Sync Updates.
5. Navigate to Lifecycle Manager > Image Depot > Vendor Addons table and make sure HPE Customization for HPE Servers 700.0.0.10.5.5-46 is displayed.
6. Edit the desired image of existing vLCM clusters, if HPE Customization for HPE Servers 700.0.0.10.5.0-108 is configured, replace it with HPE Customization for HPE Servers 700.0.0.10.5.5-46 and then save the image.