vLCM managed witness node & HMS inter-op

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
When a vSAN witness node is managed by a VLCM cluster image and there is a HMS appliance configured in the VC, then we could see the below errors due to a race condition where vLCM and HMS both try to perform some lifecycle operations on the witness hosts simultaneously.

VLCM ESXi upgrade fails with "A general system error occurred: After host 'xxxxxxxxxx' remediation was completed, the compliance check reported the host as 'non-compliant'. The image on the host does not match the image set for the cluster."

Cannot push hbr-agent VIB, the "install" task fails with "An error occurred during host configuration: /usr/sbin/esxupdate returned with exit status: 15."

When further checking esxupdate.log and lifecycle.log, two types of error messages can be seen:

KeyError: 'boot'
esximage.Errors.InstallationError: Current bootbank / bootbank is not verified and most likely a serious problem was encountered during boot, it is not safe to continue installing altbootbank. bootstate is 4, and the expected value is 0.

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 8.0.1

Cause

The root cause of this issue is a race condition between the vLCM upgrade and the HMS appliance performing a VIB push (in order to install the hbr-agent VIB) at the same time. HMS appliance is trying to make this push and install hbr-agent VIB by itself since HMS is not yet integrated with vLCM.

Resolution

Ideally, this should not happen, as the HMS appliance should not be automatically pushing VIBs onto the vSAN witness host that is managed by vLCM. For hosts that are managed by vLCM, the expectation is that the user would need to manually install the hbr-agent VIB (as opposed to the HMS appliance automatically doing this install).

Workaround:
Follow the below steps to resolve the issue:

Disable HMS appliance to auto push the hbr-agent VIBs. Steps to do this can be found below:

1. ssh to HMS VM
2. edit the config file for VR with:
/opt/vmware/hms/bin/hms-configtool -cmd reconfig -property hms-auto-install-hbragent-vib=false
3. service hms restart

Once the above is done, then you need to identify the witness VM on which the above installation error was observed, and you need to reset the VM (neither reboot the witness host nor reboot the witness ESXi DCUI, but a reset is required). This should reboot the witness host back to a state before the vLCM upgrade and race condition.
Now vLCM upgrade can be retried. And upon successfully vLCM upgrade, if the hbr-agent needs to be installed, then you can follow the instructions in this https://knowledge.broadcom.com/external/article?legacyId=87487