Stale queued resyncing object seen in vSAN cluster

Products

• VMware vSAN

Issue/Introduction

Symptom:

• "1 object in the queue (unable to provide a list of queued object)" seen from Resyncing Objects > Under Object list select "Queued" status.
  
  

Validation:

• SSH to vCenter and navigate to path "/storage/log/vmware/vsan-health/vmware-vsan-health-service.log"
  
  
• vmware-vsan-health-service.log will have below events -

2025-03-03T08:56:02.235Z INFO vsan-mgmt[08945] [VsanVcObjectSystemImpl::_GetVos opID=agw-0547688-4a19] Fetching resync query result from connected host : esxi1
2025-03-03T08:56:02.322Z INFO vsan-mgmt[08945] [VsanVcObjectSystemImpl::QuerySyncingVsanObjectsSummary opID=agw-0547688-4a19] returned resync query summary: totalObjectsToSync = 1, totalBytesToSync = 1390059454465, totalRecoveryETA = 13596, activelySyncingObjectRecoveryETA = 0, queuedForSyncObjectRecoveryETA = 13596, suspendedObjectRecoveryETA = 0, activeObjectsToSync = 0, queuedObjectsToSync = 1, suspendedObjectsToSync = 0, bytesToSyncForActiveObjects = 0, bytesToSyncForQueuedObjects = 1390059454465, bytesToSyncForSuspendedObjects = 0

• Run "esxcli vsan debug object health summary get" on vSAN node to check the status of all the objects.

Example:

[root@esxi1:~] esxcli vsan debug object health summary get

Health Status                                              Number Of Objects
---------------------------------------------------------  -----------------
remoteAccessible                                                           0
inaccessible                                                               0
reduced-availability-with-no-rebuild                                       0
reduced-availability-with-no-rebuild-delay-timer                           0
reducedavailabilitywithpolicypending                                       0
reducedavailabilitywithpolicypendingfailed                                 0
reduced-availability-with-active-rebuild                                   0
reducedavailabilitywithpausedrebuild                                       0
data-move                                                                  0
nonavailability-related-reconfig                                           0
nonavailabilityrelatedincompliancewithpolicypending                        0
nonavailabilityrelatedincompliancewithpolicypendingfailed                  0
nonavailability-related-incompliance                                       0
nonavailabilityrelatedincompliancewithpausedrebuild                        0
healthy                                                                 5223

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

• clomd service runs on every vSAN node and interacts with the vCenter to manage vSAN objects and ensure their availability. It communicates with the vCenter to receive configuration updates and reports the status of the vSAN objects.

Incase the clomd is unable to sync the object status from host to vCenter then we see a mismatch of object state from CLI and vCenter.

Resolution

• Check the /storage/log/vmware/vsan-health/vmware-vsan-health-service.log file to determine the vSAN node where the issue is occurring (ssh to vCenter and check vmware-vsan-health-service.log).
  
  
• SSH to the identified vSAN node.
  
  
• Run the following command to verify if the clomd service is running:

/etc/init.d/clomd status 

• Restart the clomd service using:

/etc/init.d/clomd restart

• After restarting the clomd service, the queued object should no longer be visible in the vSAN cluster.