vSAN Object Inaccessibility Causing VM Orphaned State Due to Ownership on Maintenance Host

Products

• VMware vSAN

Issue/Introduction

Symptoms: 

• The virtual machine reporting orphaned in vCenter server .
• This is observed after putting the host in maintenance mode. 
• It may be observed that the object inaccessible in vSAN skyline health (Cluster > Monitor > Skyline health)

Environment

vSAN 8.x 

vSAN 7.x 

Cause

Virtual machine becomes orphaned on a vSAN datastore after its namespace object becomes inaccessible.

The object was still owned by an ESXi host that had entered maintenance mode and  vSAN was unable to transfer ownership of the object to another working host.

since the host was still in  maintenance mode, The object remain inaccessible. 

This resulted in the VM becoming orphaned and inaccessible from the vSphere inventory.

Example : 

• Object status : Command output shows that the object is in inaccessible state. 

#esxcli vsan debug object health summary get 

Health Status                                              Number Of Objects
---------------------------------------------------------  -----------------
remoteAccessible                                                           0
inaccessible                                                               1
reduced-availability-with-no-rebuild                                       0

healthy                                                                  563

#esxcli vsan debug object list --health=inaccessible

Object UUID: bba03665-####-####-############

   Version: 15

   Health: inaccessible - Lost data availability.(APD)

   Owner: Host1

   Size: 0.00 GB
   Used: 1.82 GB
   Policy:
   Configuration:
      RAID_1

         Component: bba03665-4e77-####-####-############

           Component State: ABSENT,  Address Space(B): 273804165120 (255.00GB),  Disk UUID: 52f5205e-eef3-####-####-############,  Disk Name: mpx.vmhba0:C0:T7:L0:2

           Votes: 1,  Capacity Used(B): 989855744 (0.92GB),  Physical Capacity Used(B): 977272832 (0.91GB),  Host Name: Host2

         Component: 281c0368-1a01-####-####-############

           Component State: ABSENT,  Address Space(B): 273804165120 (255.00GB),  Disk UUID: 523214a2-2523-####-####-############,  Disk Name: mpx.vmhba0:C0:T6:L0:2

           Votes: 1,  Capacity Used(B): 981467136 (0.91GB),  Physical Capacity Used(B): 968884224 (0.90GB),  Host Name: Host3

      Witness: 3e906868-bea6-####-####-############

        Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: 5283dbcf-e19f-####-####-############,  Disk Name: mpx.vmhba0:C0:T7:L0:2

        Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: Host4

   Type: vmnamespace

   Path: /vmfs/volumes/vsan:79fd76100ace49d2-#############/wh-#### (Missing)

   Group UUID: bba03665-3e22-####-####-############

   Directory Name: wh-#####

Resolution

To resolve the issue, follow these steps to force vSAN to reassign object ownership:

1. Exit the affected host from maintenance mode, if it is still in maintenance.

2. Identify the UUID of the inaccessible object (e.g., the VM namespace):

   cmmds-tool find -t DOM_OBJECT -f json -u <object_uuid> | head
   

3. Find the current DOM owner (host UUID) from the output.

4. Map the owner UUID to a hostname:

   cmmds-tool find -t HOSTNAME -f json -u <owner_uuid> | grep content
   

5. SSH into the host that currently owns the object.

6. Run the following command to force the object to abdicate ownership:

   vsish -e set /vmkModules/vsan/dom/ownerAbdicate <object_uuid>
   

7. After ownership is abdicated, vSAN will reassign the object to a healthy host, restoring accessibility to the VM