Inaccessible vSAN objects flagged in Skyline health check, but no VMs are inaccessible

Products

• VMware vSAN

Issue/Introduction

There are multiple possible causes for an alert for inaccessible vSAN objects in the vSAN datastore.

KB article vSAN Health Service - Data Health - vSAN Object Health - inaccessible objects covers several scenarios, and how to address them.

 

This KB article specifically focuses on the scenario where there is an object health alert for inaccessible objects, but no VMs are inaccessible.

 

Environment

vSAN - all versions

Cause

The flagged inaccessible objects relate to VMs that have already been deleted, but a reference to each VM has been retained.

Resolution

Click on the Virtual Objects view under Monitor --> vSAN

 

 

Note the names of the flagged VMs.

- If the type is "Folder", then this is a VM namespace folder. Follow the steps below.

- If the type is "VM swap", then follow KB article vswp objects in inaccessible state to investigate this.

 

Run this command on an SSH session on one of the ESXi hosts to capture the details about the inaccessible objects.

esxcli vsan debug object list --health=inaccessible > /tmp/inaccessible_objects.txt 

 

 

Open the newly created file /tmp/inaccessible_objects.txt and inspect each VM name.

 

Sample output:

 

 

Object UUID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
   Version: 13
   Health: inaccessible - Lost data availability.(APD)
   Owner: "ESXi host name"
   Size: X GB
   Used: X GB
   Policy: ... 
   Configuration: 
      
      RAID_1
         Component: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
           Component State: ACTIVE,  Address Space(B): 0 (0.00GB),  Disk UUID: ########-####-####-####-############,  Disk Name: naa.################
           Votes: 1,  Capacity Used(B): 12582912 (0.01GB),  Physical Capacity Used(B): 4194304 (0.00GB),  Host Name: Host-A
         Component: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
           Component State: ABSENT,  Address Space(B): 0 (0.00GB),  Disk UUID: ########-####-####-####-############,  Disk Name: naa.################
           Votes: 1,  Host Name: Host-B
         Witness: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
           Component State: ABSENT,  Address Space(B): 0 (0.00GB),  Disk UUID: ########-####-####-####-############,  Disk Name: naa.################
           Votes: 1,  Host Name: Host-C

 

   Type: vmnamespace
   Path: /vmfs/volumes/vsan:XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/"VM name" (Missing)

 

Note the "Object UUID" for each missing VM namespace.

 

Check which ESXi host owns these objects:

cmmds-tool find -t DOM_OBJECT -f json -u <object_uuid> | head

 

Note the owner UUID.

{
"entries":
[
{
"uuid": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
"owner": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
"health": "Healthy",
"revision": "16",
"type": "DOM_OBJECT",
"flag": "2",

 

 

Identify which ESXi host is the owner:

cmmds-tool find -t HOSTNAME -f json -u "Owner UUID form above output" |grep content

 

 

It is likely that all of the inaccessible namespace folders from previously deleted VMs will be owned by the same ESXi host, which is holding a reference to them.

Put this ESXi host into Maintenance Mode using the "Ensure Data Accessibility" option, and then reboot it.

 

If this does not resolve the problem, please open a support case with Broadcom to investigate this further.