SDDC Host Update fails "Message: Cannot invoke "java.util.List.iterator()" because 'customlmageSpecList" is null." // "Cannot invoke "java.util.List.iterator()" because "customlmageSpecList" is null."

Products

• VMware vSphere ESXi

Issue/Introduction

Upgrading Cluster via SDDC downloaded bundle can fail with the following error 

Upgrade DOMAIN - [<Domain name>] using BUNDLE - HOST:<Build ID>
Upgrade - ESX_HOST <HOSTNAME>    FAILED
Description                      Upgrade - ESX_HOST <Hostname>
Process Messages                 Cannot invoke "java.util.List.iterator()" because 
                                 "customlmageSpecList" is null.
Error                            Message: Cannot invoke "java.util.List.iterator()" because
                                 'customlmageSpecList" is null.
Remediation Message:             Cannot invoke "java.util.List.iterator()" because
                                 'customlmageSpecList" is null
                                 Reference Token: GKUQOK


Logs domainmanager.log 

YYYY-MM-DDTHH:MM:SS.<> INFO  [vcf_lcm,0000000000000000,0000,upgradeId=95ac130e-790f-48ff-a7de-997ee2a6cc55,resourceType=ESX_HOST,resourceId=268455d5-0878-4c97-9d66-41a306dcdf00,bundleElementId=4e5d3fc2-a467-4a9f-a5a7-de4de361c585] 
[c.v.e.s.l.p.i.e.EsxVumPrimitiveImpl,Upgrade-7] ESX custom image upgrade spec is {}
YYYY-MM-DDTHH:MM:SS.<> ERROR [vcf_lcm,0000000000000000,0000] [c.v.e.s.l.orch.PrimitiveServiceImpl,Upgrade-7] primitive post upgrade returned exception
java.lang.NullPointerException: Cannot invoke "java.util.List.iterator()" because "customImageSpecList" is null
        at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxVumPrimitiveImpl.prepareEsxVumUpdateStateObjects(EsxVumPrimitiveImpl.java:353)
        at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxVumPrimitiveImpl.startOrResumeUpgrade(EsxVumPrimitiveImpl.java:244)
        at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxVumPrimitiveImpl.postUpgrade(EsxVumPrimitiveImpl.java:DD0)
        at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.postUpgradeAsync(PrimitiveServiceImpl.java:325)
        at com.vmware.evo.sddc.lcm.orch.PrimitiveServiceImpl.lambda$postUpgrade$0(PrimitiveServiceImpl.java:165)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

Environment

SDDC 5.X 

Cause

When upgrading non-vLCM clusters - i.e. no cluster image, SDDC patches using the vanilla, vSphere default, bundle.

For scenarios where customers want to patch using the additional Vendor ISO,  the following process needs to be followed
https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/vmware-cloud-foundation-lifecycle-management/upgrade-the-management-domain-to-vmware-cloud-foundation-5-2-lifecycle/upgrade-esxi-with-vsphere-lifecycle-manager-baselines-for-vmware-cloud-foundation-5-2-lifecycle/upgrade-esxi-with-custom-iso-lifecycle.html  


The alarm/ event is triggered even if the config looks ok 
# grep -i "lcm.esx.upgrade.custom.image.spec" application-prod.properties
lcm.esx.upgrade.custom.image.spec=/nfs/vmware/vcf/nfs-mount/esx-custom-image-upgrade-spec.json

We need to check the  esx-custom-image-upgrade-spec.json for the path to the Custom ISO.
Log means the Custom ISO path is null
          [c.v.e.s.l.p.i.e.EsxVumPrimitiveImpl,Upgrade-7] ESX custom image upgrade spec is {}

Resolution

Correct ESX custom image JSON using the following template aka  esx-custom-image-upgrade-spec.json

 {
"esxCustomImageSpecList": [{
"bundleId": "bundle ID of the ESXi bundle you downloaded",
"targetEsxVersion": "ESXi version for the target VMware Cloud Foundation version",
"useVcfBundle": false,
 "domainId": "xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx",
 "clusterId": "xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx",
"customIsoAbsolutePath": "Path_to_custom_ISO"
}]
}

Optional Skip the configuration 
# grep -i "lcm.esx.upgrade.custom.image.spec" application-prod.properties
lcm.esx.upgrade.custom.image.spec=

The applied host profile will be the vSphere default patch only, there will be no Vendor components updated. 

Additional Information

https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-5-2-and-earlier/5-2/vmware-cloud-foundation-lifecycle-management/upgrade-the-management-domain-to-vmware-cloud-foundation-5-2-lifecycle/upgrade-esxi-with-vsphere-lifecycle-manager-baselines-for-vmware-cloud-foundation-5-2-lifecycle/upgrade-esxi-with-custom-iso-lifecycle.html