ESXi upgrade from SDDC Manager fails at stage: ESX Upgrade VUM Stage Scan Host task

Products

VMware SDDC Manager VMware Cloud Foundation

Issue/Introduction

Unable to Patch or Upgrade ESXi Host part of Workload Domain in VCF Environment, Error: ESX Upgrade VUM Stage Scan Host task Failed
ESXi hosts installed using a custom ISO cannot be patched or upgraded successfully using the ESXi bundles provided in the SDDC's Bundle Management section

Error Observed in SDDC Manager UI

ESX Upgrade VUM Stage Scan Host task Failed

Error Onserved in vCenter by Expanding the Scan Entity Task for the Failed Host

The upgrade has VIBs that are missing dependencies. Remove the VIBs or use Image Builder to create a custom upgrade ISO image that contains the missing dependencies, and try to upgrade again

Environment

VMware Cloud Foundation

Cause

When ESXi Host is installed using Custom Image there are a few OEM Bundle/vibs contained in it which are not present in Vanilla Image leading to Patch/Upgrade failure

Resolution

Download the appropriate vendor-specific image for the Custom ISO for the desired build being used to update/upgrade on a computer with internet access by following How to download ISO using Broadcom Support Portal.
If no vendor-specific ISO is available for the required version of ESXi, then you can create one. See Create a Custom ISO Image for ESXi.
SSH into the SDDC Manager using the vcf user and then su root
Create a directory for the vendor iso under the /nfs/vmware/vcf/nfs-mount directory.
```
mkdir /nfs/vmware/vcf/nfs-mount/esx-upgrade-partner-binaries
```
Copy the vendor-specific iso to the directory you created on the SDDC Manager appliance. For example, you can copy the iso to the /nfs/vmware/vcf/nfs-mount/esx-upgrade-partner-binaries directory.

Change permissions on the directory where you copied the image.

chmod -R 775 /nfs/vmware/vcf/nfs-mount/esx-upgrade-partner-binaries/

Change owner to vcf.

chown -R vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/esx-upgrade-partner-binaries/

Create a JSON file as esx-custom-image-upgrade-spec.json in the /nfs/vmware/vcf/nfs-mount
Note, If the JSON file is not saved in the correct directory, the stock VMware Cloud Foundation Image is used for the upgrade and the custom VIBs are overwritten.

Create an ESX custom image JSON using the following template.

{
"esxCustomImageSpecList": [{
"bundleId": "bundle ID of the ESXi bundle available in Bundle Management Section of SDDC Manager",
"targetEsxVersion": "ESXi version for the target VMware Cloud Foundation version",
"useVcfBundle": false,
 "domainId": "########-####-####-########",
 "clusterId": "########-####-####-########",
"customIsoAbsolutePath": "Path_to_image_of_custom_ISO"
}]
}

where

Parameter	Description and Example Value
bundleId	ID of the Vanilla ESXi upgrade bundle downloaded through Bundle Management of SDDC. You can retrieve the bundle ID by navigating to the Lifecycle Management > Bundle Management page and clicking View Details to view the bundle ID. For example, 8c0de63d-b522-4db8-be6c-f1e0ab7ef554 . Note, ID for Custom Image uploaded will not be shown in Bundle Managment hence Bundle ID of ESXi upgrade bundle downloaded through Bundle Management of SDDC is required to be used here
targetEsxVersion	Version of the ESXi bundle you downloaded. You can retrieve the target ESXi version by navigating to the Lifecycle Management > Bundle Management page and clicking View Details to view the "Update to Version".
useVcfBundle	Specifies whether the VMware Cloud Foundation ESXi bundle is to be used for the upgrade. If you want to upgrade with a custom image, ensure that this is set to false.
domainId (optional, VCF 4.4.1.1 and later only)	ID of the specific workload domain where the custom image needs to be applied. Use the VMware Cloud Foundation API (GET /v1/domains) to get the IDs for your workload domains.
clusterId (optional, VCF 4.4.1.1 and later only)	ID of the specific cluster within a workload domain to apply the custom image. If you do not specify a clusterId, the custom image will be applied to all clusters in the workload domain. Use the VMware Cloud Foundation API (GET /v1/clusters) to get the IDs for your clusters.
customIsoAbsolutePath	Path to the custom image file on the SDDC Manager appliance For example, /nfs/vmware/vcf/nfs-mount/esx-upgrade-partner-binaries/VMware-VMvisor-Installer-7.0.0.update01-17325551.x86_64-DellEMC_Customized-A01.iso Note, Please make sure the path is correct else the upgrade will proceed with the VMware Cloud Foundation stock image bundle and replace the custom VIBs in your environment with the stock VIB's or the activity may fail with same vib dependency error.

Here is an example of a completed JSON template.

{
"esxCustomImageSpecList": [{
"bundleId": "8c0de63d-b522-4db8-be6c-f1e0ab7ef554",
"targetEsxVersion": "8.0.1-########",
"useVcfBundle": false,
"customIsoAbsolutePath":
"/nfs/vmware/vcf/nfs-mount/esx-upgrade-partner-binaries/VMware-VMvisor-Installer-8.0.0.update01-########.x86_64-DellEMC_Customized-A01.iso"
}]
}

If there are more than 1 cluster which needs to be updated with Custom Image then change the JSON template to include entries for the other cluster too in the format:

For Example if you have 2 Cluster one is Dell and other one is HPE Cluster then the esx-custom-image-upgrade-spec.json should look like:

{
   "esxCustomImageSpecList": [
       {
           "bundleId": "aa7b16b1-########-9ced-########",
           "targetEsxVersion": "8.0.2-########",
           "useVcfBundle": false,
           "domainId": "########-####-####-########",
           "clusterId": "########-####-####-########",
           "customIsoAbsolutePath": "/nfs/vmware/vcf/nfs-mount/esx-upgrade-partner-binaries/VMware-ESXi-8.0.2-23305546-DELL.iso"
       },
       {
           "bundleId": "aa7b16b1-########-44b7-9cfd-########",
           "targetEsxVersion": "7.0.1-########",
           "useVcfBundle": false,
           "domainId": "########-####-####-########",
           "clusterId": "########-####-####-########",
           "customIsoAbsolutePath": "/nfs/vmware/vcf/nfs-mount/esx-upgrade-partner-binaries/VMware-ESXi-7.0.2-17867351-HP.iso"
       }
   ]
}

Set the correct permissions on the /nfs/vmware/vcf/nfs-mount/esx-custom-image-upgrade-spec.json file

chmod -R 775 /nfs/vmware/vcf/nfs-mount/esx-custom-image-upgrade-spec.json
chown -R vcf_lcm:vcf /nfs/vmware/vcf/nfs-mount/esx-custom-image-upgrade-spec.json

Open the /opt/vmware/vcf/lcm/lcm-app/conf/application-prod.properties file and add the path of the JSON template created in Step # 8 in lcm.esx.upgrade.custom.image.spec
```
lcm.esx.upgrade.custom.image.spec=/nfs/vmware/vcf/nfs-mount/esx-custom-image-upgrade-spec.json
```
Restart LCM service
```
systemctl restart lcm
```
In SDDC Manager UI, click on Workload Domains.
Click the domain you want to upgrade and then click the Updates/Patches tab.
Schedule the ESXi upgrade bundle.
Monitor the upgrade progress. See Monitor VMware Cloud Foundation Updates.
After the upgrade is complete, confirm the ESXi version by clicking Current Versions. The ESXi hosts table displays the current ESXi version.

Additional Information

Please reach out to Broadcom Support in case or any question or clarification needed while following the KB