ESXi hosts automatically enter and exit maintenance mode post vSphere Replication upgrade

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

vSphere Replication 8.7 or later versions upgraded.
ESXi hosts enters and exits maintenance mode to update hbr-agent VIB
This is a rolling placement of hosts in maintenance mode and thus results in migrating VMs out of a host that enters maintenance mode.

In some scenarios, this automated process may ignore vLCM cluster-level remediation settings (such as parallel remediation limits) and attempt to place multiple hosts into maintenance mode simultaneously. This can exhaust cluster compute capacity, leading to DRS errors such as: 'failed getting host recommendation from drs to enter maintenance mode from cluster... reason: There are no active hosts in the cluster.'

Validation:

In /var/log/vmware/vum-server/vmware-vum-server.log from vCenter VLCM checks for the host configuration before sending instructions to place ESXi host in maintenance mode

YYYY-MM-DDTHH:MM:57.062Z info vmware-vum-server[1408796] [Originator@6876 sub=EHP opID=f4c10152-2332-47cf-b081-##########] CheckContext: {entityMoId: "host-####", vapiSession: "###########", env: {"Host part of VMC": false, "vLCM-VMC integration, Pod service enabled": false, }}, HostCheckContext: {spec: {{ com.vmware.esx.health.hosts.check_spec : { evacuation_action : Optional< >, exclude_checks : [ ] , hosts : Optional< >, maintenance_mode_type : Optional< >, memory_reservation : Optional< >, perspective : BEFORE_MODIFICATION, target_spec : Optional< {{ com.vmware.esx.health.hosts.target_spec : { state_changes : [ {{ map-entry : { key : VMware-HBR-Agent, value : UPGRADE, } }} , ] , } }} >, upgrade_actions : Optional< >, vsan_streched_cluster : Optional< domain-xxxxx>, } }} }

YYYY-MM-DD THH:MM:21.903Z info vmware-vum-server[614236] [Originator@6876 sub=ClusterApplySolutionTask] [Task, 524] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplySolutionTask ID:########-####-###-####-###########. Applying (solution = VMware-HBR-Agent) solution - Start batch (batch count = 1)
YYYY-MM-DDTHH:MM:21.903Z info vmware-vum-server[614236] [Originator@6876 sub=ClusterApplySolutionTask] [Task, 524] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplySolutionTask ID:########-####-###-####-###########. Host Apply Solution - Check Before EnterMaintenanceMode - (cluster=domain-##) - (host=host-##) - (solution=VMware-HBR-Agent)
YYYY-MM-DDTHH:MM:21.903Z info vmware-vum-server[614236] [Originator@6876 sub=ClusterApplySolutionTask] [Task, 524] Task:com.vmware.vcIntegrity.lifecycle.ClusterApplySolutionTask ID:########-####-###-####-###########. CheckBeforeEnterMaintenanceMode (solution = VMware-HBR-Agent) on (host = host-##) - Solution Component Upgrade - enter MaintenanceMode

In /var/run/log/hostd.log host enter and exit maintenance mode events are observed before and after vib install

YYYY-MM-DDTHH:MM:57.020Z In(166) Hostd[2100975]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=######## sid=######## user=dcui:vsanmgmtd] Event 150847 : Host .com in ha-datacenter has entered maintenance mode

YYYY-MM-DDTHH:MM:28.855Z In(166) Hostd[2100971]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 150960 : The host has exited maintenance mode

Environment

vSphere Replication 8.7

vSphere Replication 8.8

vSphere Replication 9.x

Cause

ESXi hosts entering maintenance mode is triggered by VLCM (VMware Life Cycle Manager) which then pushes hbr-agent VIB to these hosts post vSphere Replication appliance upgrade.
This is an expected behavior and as per the design.

Resolution

This issue has been fixed in VMware Live Recovery Appliance 9.0.4 & VMware ESX 9.0.1.0 that is part of VMware Cloud Foundation 9.0

ESX hosts automatically enter and exit maintenance mode

If you use VMware Life Cycle Manager (vLCM), after upgrading vSphere Replication, the ESX hosts automatically enter and exit maintenance mode.

This issue is fixed in the VMware Live Recovery 9.0.4 appliance.

Active Mitigation / EMERGENCY STOP

If the automated remediation loop is actively running and causing service disruption:

Immediately stop the HMS service on the Replication Appliance to sever the connection to vCenter:

systemctl stop hms
Log into the vSphere Client and manually cancel any pending or stuck Apply Solution, Remediate Cluster, or Enter Maintenance Mode tasks.
Apply the configuration change outlined in Workaround 1.
Start the HMS service:

systemctl start hms

This behavior is being reviewed and being worked on by Broadcom Engineering for an enhancement.

Workaround 1:

Disable automatic VIB installation/update by vSphere Replication:

1. SSH to the vSphere Replication appliance and run one of the commands below depending on the version of replication appliance:

Run the command:

/opt/vmware/hms/bin/hms-configtool -cmd reconfig -property hms-auto-install-hbragent-vib=false

2. Manually install the VIBs on each ESXi host:

For environments using vLCM Image Mode, you must extract the offline bundle (VMware-HBR-Agent-xxx.zip) from the vSphere Replication ISO, import it into the vLCM Update repository, and add it to your Cluster Image as an independent component. You can then remediate the cluster safely using standard vLCM procedures.

For full step-by-step instructions, see: [How to manage vSphere replication solution in cluster image manually]

3. NEW: Restart the Host Management Service (HMS) to commit the changes to memory:

systemctl restart hmsHow to manage vSphere replication solution in cluster image manually

Workaround 2:

Please schedule vSphere Replication upgrades during non-business hours, as placing hosts in Maintenance Mode will trigger vMotion, potentially impacting workload performance.

Additional Information

This issue is not observed when running on VCF 4.x or VCF 5.x since VMware Update Manager manages the updates and it uses Baseline mode
This issue is not observed even after moving to vLCM, but if you are still using Baseline mode - However, you will receive alerts on the vCenter server and vSAN Skyline health, prompting to move to image mode