Error during upgrade of vSAN disk format "Failed to convert object(s) on vSAN."
search cancel

Error during upgrade of vSAN disk format "Failed to convert object(s) on vSAN."

book

Article ID: 367420

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

  • vSAN disk format upgrade fails with the below error on "vCenter UI > vSAN cluster > Configure > Disk management"

Task Name - Convert disk format for vSAN
Status - General vSAN error. Failed to convert object(s) on vSAN.

  • Disk format upgrade may fail when triggered from vCenter UI , vCenter cli and SDDC manager.

  • On-disk format version for all the disks shows its upgraded and running on upgraded version.

Environment

VMware vSAN (All Versions)

Cause

On-disk format version upgrade failed due to the stale stuck object with older object format version with no DOM owner.

The below command will list the object format versions and the number of objects in these version(s):

# esxcli vsan debug object list --all | grep -i version | sort | uniq -c

Note: The command will take a while to run. For 8.x '--all' is not working as expected so use --max-number=<number higher than the amount of objects in the environment>


Below is an example of the object structure of a stale vSAN object:

  •  There is no DOM owner  for the object "aaaaaaaa-1111-bbbb-2222-cccccccccccc".

[root@host#: esxcli vsan debug object list -u aaaaaaaa-1111-bbbb-2222-cccccccccccc
Object UUID: aaaaaaaa-1111-bbbb-2222-cccccccccccc
   Version: 15
   Health: N/A
   Owner: 00000000-0000-0000-0000-000000000000
   Size: 0.00 GB
   Used: 0.00 GB
   Used 4K Blocks: 0.00 GB
   Policy:
   Configuration:

Steps to verify the stale object:

  • Failed to get the object attributes as they are not found.

[root@host#:~] /usr/lib/vmware/osfs/bin/objtool getAttr -u aaaaaaaa-1111-bbbb-2222-cccccccccccc
Failed to get object attributes : No such file or directory 131076.
object getAttr error: Failure

  • DOM owner is not available and the object reports unhealthy.

[root@host#:~] cmmds-tool find -t DOM_OBJECT -u aaaaaaaa-1111-bbbb-2222-cccccccccccc

owner=00000000-0000-0000-0000-000000000000(Health: Unhealthy) uuid=aaaaaaaa-1111-bbbb-2222-cccccccccccc type=DOM_OBJECT rev=0 minHostVer=3  [content = ("Configuration" (("CSN" l8) ("addressSpace" l1099511627776) ("scrubStartTime" l+1743381142212690) ("objectVersion" i15) ("highestDiskVersion" i15) ("muxGroup" l798880971142367392) ("groupUuid" ########-####-####-##############) ("raidFact" i200) ("subClusterUUID" ########-####-####-##############) ("compositeUuid" aaaaaaaa-1111-bbbb-2222-cccccccccccc) ("objCapabilities" i1)) ("Concatenation" (("scope" i1)) ("RAID_1" (("scope" i3)) ("Component" (("capacity" (l0 l273804165120)) ("addressSpace" l273804165120) ("componentState" l5) ("componentStateTS" l1743381142) ("faultDomainId" ########-####-####-##############) ("nVotes" i5) ("lastScrubbedOffset" l273804165120) ("subFaultDomainId" ########-####-####-##############) ########-####-####-############## 

  • Since this object is stale, it does not have a health state to report, hence all the objects in the vSAN cluster appear to be healthy.

[root@host#:~] cmmds-tool find -t DOM_OBJECT -u aaaaaaaa-1111-bbbb-2222-cccccccccccc

# esxcli vsan debug object health summary get
Health Status                                              Number Of Objects
---------------------------------------------------------  -----------------
remoteAccessible                                                           0  
inaccessible                                                               0  
reduced-availability-with-no-rebuild                                       0  
reduced-availability-with-no-rebuild-delay-timer                           0  
reducedavailabilitywithpolicypending                                       0  
reducedavailabilitywithpolicypendingfailed                                 0  
reduced-availability-with-active-rebuild                                   0  
reducedavailabilitywithpausedrebuild                                       0  
data-move                                                                  0  
nonavailability-related-reconfig                                           0  
nonavailabilityrelatedincompliancewithpolicypending                        0  
nonavailabilityrelatedincompliancewithpolicypendingfailed                  0  
nonavailability-related-incompliance                                       0  
nonavailabilityrelatedincompliancewithpausedrebuild                        0  
healthy                                                                  854

 

  • In /var/run/log/vsansystem.log we can see that the stale object was not found in CMMDS ( Cluster Membership, Monitoring, and Directory Services).

2026-02-17T02:21:58.005Z Wa(164) vsansystem[40370874]: [vSAN@6876 sub=Libs opId=####] Object aaaaaaaa-1111-bbbb-2222-cccccccccccc not found in CMMDS with error Not found
2026-02-17T02:21:58.005Z Er(163) vsansystem[40370874]: [vSAN@6876 sub=CMMDSUtil opId=####] Failed to check accessibility state for object 'aaaaaaaa-1111-bbbb-2222-cccccccccccc': Not found

  • We can see the below entries in the var/run/log/epd.log as well:

202##### 2099341 info epd EPDPollPeriodicCrawlerCB: Probably stale DISCARDED_COMPONENTS entry ########-####-####-############# detected by the crawler and scheduled for unpublishing.

Resolution

Stale vSAN object needs to be deleted in order to proceed with the vSAN disk format upgrade.

Please contact Broadcom Support for further assistance to validate the stale object and proceed with object deletion.

Powered by Wolken Software