vCLS VMs are not getting Deployed Automatically after a vSAN Cluster Shutdown.
search cancel

vCLS VMs are not getting Deployed Automatically after a vSAN Cluster Shutdown.

book

Article ID: 322139

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSAN VMware vSphere ESXi

Issue/Introduction

Symptoms:

vCLS VMs are not getting deployed automatically on a cluster after a vSAN Cluster Shutdown.

While looking into the ESX Agent Manager we can see the status as uninstalled.

  • From the EAM Logs we can see below:
2023-08-07T05:55:50.351Z |  INFO | vim-monitor | ClusterVMAgency.java | 1067 | ClusterVMAgency(ID:'Agency:6e53c80d-c040-4c83-bc17-45dea5abcded:null') attaching to ClusterComputeResource:domain-c8
2023-08-07T05:56:00.877Z |  INFO | vim-monitor | ClusterVMAgency.java | 1067 | ClusterVMAgency(ID:'Agency:3f770b8f-f31e-4953-a10b-45dea5ijklmno:null') attaching to ClusterComputeResource:domain-c2013
2023-08-07T05:56:03.951Z |  INFO | vim-monitor | ClusterVMAgency.java | 1067 | ClusterVMAgency(ID:'Agency:f6218f9f-b458-49af-a1d3-70dfda0mnopq:null') attaching to ClusterComputeResource:domain-c2102
  • Looking at the logs destroyAgency API call was probably made, because the goal state of the agency changed from enabled to uninstalled.
2023-08-07T10:26:17.462Z |  INFO | vlsi | AgencyBase.java | 1375 | Goal state change from enabled to uninstalled (ClusterVMAgency(ID:'Agency:3f770b8f-f31e-4953-a10b-45dea5ijklmno:null'))
2023-08-07T10:26:17.468Z |  INFO | vlsi | ClusterAgent.java | 613 | Goal state changed from enabled to uninstalled (ClusterAgent(ID: 'Agent:1ccee090-4796-4695-b023-1f028f531cbe:null'))
2023-08-07T10:26:17.497Z |  INFO | vlsi | AgencyBase.java | 1375 | Goal state change from enabled to uninstalled (ClusterVMAgency(ID:'Agency:f6218f9f-b458-49af-a1d3-70dfda0mnopq:null'))
  • It seems like the EAM couldn't finish the agency destruction and continued the agency destruction after it:
2023-08-08T03:16:52.735Z |  INFO | vim-monitor | ClusterVMAgency.java | 1067 | ClusterVMAgency(ID:'Agency:6e53c80d-c040-4c83-bc17-45dea5abcded:null') attaching to ClusterComputeResource:domain-c8
2023-08-08T03:16:55.126Z |  INFO | vim-monitor | AgencyBase.java | 1509 | ClusterVMAgency(ID:'Agency:6e53c80d-c040-4c83-bc17-45dea5abcded:null') changed status from green to yellow
2023-08-08T03:16:55.152Z |  INFO | vim-monitor | ClusterVMAgency.java | 1067 | ClusterVMAgency(ID:'Agency:f6218f9f-b458-49af-a1d3-70dfda0mnopq:null') attaching to ClusterComputeResource:domain-c2102
2023-08-08T03:16:55.898Z |  INFO | vim-monitor | AgencyBase.java | 1509 | ClusterVMAgency(ID:'Agency:f6218f9f-b458-49af-a1d3-70dfda0mnopq:null') changed status from green to yellow
2023-08-08T03:16:55.922Z |  INFO | vim-monitor | ClusterVMAgency.java | 1067 | ClusterVMAgency(ID:'Agency:3f770b8f-f31e-4953-a10b-45dea5ijklmno:null') attaching to ClusterComputeResource:domain-c2013
2023-08-08T03:16:56.826Z |  INFO | vim-monitor | AgencyBase.java | 1509 | ClusterVMAgency(ID:'Agency:3f770b8f-f31e-4953-a10b-45dea5ijklmno:null') changed status from green to yellow
2023-08-08T07:17:08.581Z |  INFO | vim-monitor | ClusterVMAgency.java | 1067 | ClusterVMAgency(ID:'Agency:6e53c80d-c040-4c83-bc17-45dea5abcded:null') attaching to ClusterComputeResource:domain-c8
2023-08-08T07:17:10.934Z |  INFO | vim-monitor | AgencyBase.java | 1509 | ClusterVMAgency(ID:'Agency:6e53c80d-c040-4c83-bc17-45dea5abcded:null') changed status from green to yellow
2023-08-08T07:17:10.978Z |  INFO | vim-monitor | ClusterVMAgency.java | 1067 | ClusterVMAgency(ID:'Agency:f6218f9f-b458-49af-a1d3-70dfda0mnopq:null') attaching to ClusterComputeResource:domain-c2102
2023-08-08T07:17:11.510Z |  INFO | vim-monitor | ClusterVMAgency.java | 1067 | ClusterVMAgency(ID:'Agency:3f770b8f-f31e-4953-a10b-45dea5ijklmno:null') attaching to ClusterComputeResource:domain-c2013

Environment

VMware vSAN 8.0.x
VMware vCenter Server 8.0.x
VMware vSAN 7.0.x
VMware vCenter Server 7.0.x

Cause

The cause of the agency having a green status with a uninstalled goal state with no agent vCLS VMs is because of the way EAM handles partially destroyed agencies after a service restart.

2023-08-08T03:22:56.128Z | INFO | vlsi | LocalizationFilter.java | 108 | API COMPLETE: ClusterVMAgency(ID:'Agency:3f770b8f-f31e-4953-a10b-45dea5ijklmno:null').queryRuntime[opId=3288210, sessionId=0DAEEE8B]. Result:eam.EamObject.RuntimeInfo {  issue = <unset>,  goalState = 'uninstalled',  entity = 'Agency:3f770b8f-f31e-4953-a10b-45dea5ijklmno:null',  status = 'green',

And because there are no agent VMs present, the DRS is not working on the cluster.

Resolution

NOTICE: Before proceeding with vCenter mob modification, please take snapshot of the vCenter. If the vCenters are in enhanced linked mode, offline snapshot of all the linked vCenters is required.  Refer the instructions in the KB VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice for more information.

1. Restart eam service. Refer the instuctions on KB Stopping, Starting or Restarting VMware vCenter Server Appliance 6.x & above services

service-control --restart eam

2. Go to browser and use the vCenter URL https://<vcsa-dns-name>/eam/mob/?moid=<agency_id>

<vcsa-dns-name> is the name of the VC.

<agency_id> is the ID of the agency that needs to be destroyed. In this case it is 3f770b8f-f31e-4953-a10b-45dea5ijklmno

3. Click DestroyAgency.

4. A new window will be opened and there you have to click "Invoke Method".

5. When the agency is destroyed, the vCLS service will detect that and will tell EAM to create a new agency in the cluster with agent VMs.

If further assistance needed on this issue, please contact Broadcom Support.

Additional Information

Impact/Risks:

There are no agent VMs present, the DRS is not working on the cluster.

Powered by Wolken Software