vSphere HA issues after patching/updating vCenter

Products

• VMware vCenter Server

Issue/Introduction

• After a vCenter upgrade, the VUM/vLCM service is restarted. At this time, vCenter will deploy a new version of HA to the ESXI hosts using vLCM.
• This operation may fail. The following log may contain similar entries: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-##.log 

2023-03-17T14:23:01.962Z info vmware-vum-server[08406] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.SetSolutionTask] [Task, 486] Task:com.vmware.vcIntegrity.lifecycle.SetSolutionTask ID:########-####-####-####-########2a59. Finalizing Task

2023-03-17T14:23:01.962Z info vmware-vum-server[08406] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.SetSolutionTask] [Task, 486] Task:com.vmware.vcIntegrity.lifecycle.SetSolutionTask ID:########-####-####-####-########2a59. Task Finalization completed.

2023-03-17T14:23:01.962Z error vmware-vum-server[08406] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.SetSolutionTask] [Task, 486] Task:com.vmware.vcIntegrity.lifecycle.SetSolutionTask ID:########-####-####-####-########2a59. Task Failed. Error: Error:
--> com.vmware.vapi.std.errors.invalid_argument
--> Messages:
--> com.vmware.vcIntegrity.lifecycle.softwarespec.InvalidImage<Image is not valid.>
--> 

• HA agent unreachable
• When attempting to enable vSphere HA, the following error may also appear:
  
  Cannot install the vCenter Server agent service. "Unknown installer error"
  
  
• Enabling vSphere HA task hangs
• vSphere HA Agent is in the Uninitialized State
• The issue is intermittent.

Environment

VMware vCenter Server 8.0.x

Cause

At VUM/vLCM service restart, the ImageService validates/loads VMware online depots and any other online depots that have been added along with loading the HA/WCP depots from vCenter to vLCM.  Due to the number and/or size of the online depots, validating/loading them needs time. This may block HA depot load which needs the write lock of depot cache by the depot validation operation which holds the read lock or the load operation which holds thee write lock.

On the other hand, if HA upgrade operation comes early before the depot loads finish. This operation validates the desired image with the new version of HA component.  However, as the depot is not ready yet, the new version of HA component is not found. Thus, software validation fails with error mentioned  above.

Resolution

This is the behavior for multiple releases. Due to the accumulation of base images and add-ons, depot validation/load becomes heavier so the issue becomes more often.

Workaround:

Disable/enable HA to fix the issue.