Transient "Agent status changed" warning during vSphere 8.0 U3 cluster upgrade

Products

• VMware vSphere ESXi 8.0
• VMware vCenter Server 8.0

Issue/Introduction

When performing a rolling upgrade of an ESXi cluster to 8.0 U3, an "Agent status changed" warning alarm can be triggered under "All Issues" in the vSphere Client.

This warning typically manifests immediately after the first ESXi host in the cluster is successfully upgraded to 8.0 U3 and removed from maintenance mode.
During this temporary mixed-version state, the remaining non-upgraded hosts can trigger an alarm.

The following log entries are recorded in /var/log/vmware/eam/eam.log:

YYYY-MM-DDThh:mm:ss.###Z |  INFO | vim-inv-update | VcHostSystem.java | ### | VcHostSystem(ID: host-##) is out of maintenance mode.
YYYY-MM-DDThh:mm:ss.###Z |  INFO | vim-inv-update | ClusterAgent.java | ### | [checkGoal:ClusterAgent(ID: ########-####-####-####-############)] enabled.
YYYY-MM-DDThh:mm:ss.###Z |  INFO | vim-inv-update | AgentBase.java | ### | ClusterAgent(ID: ########-####-####-####-############) changed status from green to yellow
...
YYYY-MM-DDThh:mm:ss.###Z |  INFO | vim-inv-update | IssueHandlerBase.java | ### | Updating issues:
New issues:
 [
eam.issue.cluster.agent.VmPoweredOff {
   description = <unset>,
   time = YYYY-MM-DD hh:mm:ss,###,
   key = 4,
   agency = 'Agency:########-####-####-####-############:null',
   agencyName = 'vCLS',
   solutionId = 'VSPHERE.LOCAL\vpxd-extension-########-####-####-####-############',
   solutionName = ' ',
   cluster = 'ClusterComputeResource:domain-c#:########-####-####-####-############',
   agent = 'Agent:########-####-####-####-############:null',
   vm = 'VirtualMachine:vm-##:########-####-####-####-############',                                                                                                                       }
]
Removed issues: []

Environment

vSphere 8.0 U3

Cause

This behavior is an expected, transient side-effect of the automatic migration to the new Embedded vCLS architecture introduced in vSphere 8.0 U3.

When the first ESXi 8.0 U3 host becomes active in the cluster, vCenter Server automatically initiates the decommissioning of the legacy vCLS VMs (which were running on the old ESXi).
This does not indicate a functional failure of DRS or HA.

Resolution

This alarm during upgrade can be safely ignored.

If necessary, follow these steps to clear the alarm:

1. In the vSphere Client, navigate to the cluster [Monitor] -> [vSphere Cluster Services] -> [Health] tab.
2. Confirm that the "Cluster Services Status" has successfully transitioned to "Healthy".
3. If the status is Healthy, the new Embedded vCLS is active on the 8.0 U3 host.
   Manually clear the "Agent status changed" warning by selecting "Reset to Green".

Additional Information