Symptoms:
When a vSAN witness node is managed by a VLCM cluster image and there is a HMS appliance configured in the VC, then we could see the below errors due to a race condition where vLCM and HMS both try to perform some lifecycle operations on the witness hosts simultaneously.
VLCM ESXi upgrade fails with "A general system error occurred: After host 'xxxxxxxxxx' remediation was completed, the compliance check reported the host as 'non-compliant'. The image on the host does not match the image set for the cluster."
- Cannot push hbr-agent VIB, the "install" task fails with "An error occurred during host configuration: /usr/sbin/esxupdate returned with exit status: 15."
When further checking esxupdate.log and lifecycle.log, two types of error messages can be seen:
- KeyError: 'boot'
- esximage.Errors.InstallationError: Current bootbank / bootbank is not verified and most likely a serious problem was encountered during boot, it is not safe to continue installing altbootbank. bootstate is 4, and the expected value is 0.