vSAN Health Service - Data Health - vSAN Object Health - inaccessible objects

Products

• VMware vSAN

Issue/Introduction

vSAN Health Service - Data Health - vSAN Object Health reports inaccessible objects

In addition, one or more of the following Symptoms do apply:

• ESXi host which is part of vSAN cluster will fail to enter maintenance mode using either "Ensure Accessibility" or "Full Data Migration" mode selected when inaccessible objects are present.
• vSAN on-disk format (ODF) upgrade will fail as the objects are not in a healthy state.
• VMs will report as inaccessible/invalid in vCenter Web Client
• VMs are inaccessible state. You tried to unregister and re-register with no luck

Environment

VMware vSAN (All Versions)

Cause

vSAN objects become inaccessible when there are not enough Active components due to the loss of Fault Domains that exceed the storage policy's Failures to Tolerate to keep the object accessible.

• This can happen if hosts are in vSAN Decom/ESXi maintenance mode
• Failed disks/disk groups
• Network partition
• vSAN cluster shutdown/restart is in an inconsistent state

Resolution

1. Follow KB vSAN Health Service - Data Health – vSAN Object Health and purge any possible inaccessible vswp (VM Swap file) objects by clicking "Purge inaccessible VM Swap Objects" as these files are safe to delete. This option will detect all the inaccessible vswp objects and give you an option to purge them. These inaccessible VM Swap Objects can be removed without risk to production.
   
   
2. Check to see if any hosts are in vSAN Decom state by running the below command:
   
   echo "hostname,decomState,decomJobType";for host in $(cmmds-tool find -t HOSTNAME -f json |grep -B2 Healthy|grep uuid|awk -F \" '{print $4}');do hostName=$(cmmds-tool find -t HOSTNAME -f json -u $host|grep content|awk -F \" '{print $6}');decomInfo=$(cmmds-tool find -t NODE_DECOM_STATE -f json -u $host |grep content|awk '{print $3 $5}'|sed 's/,$//');echo "$hostName,$decomInfo";done|sort
   
   Example output:
   hostname,  decomState,  decomJobType
   host-21,0,0
   host-22,0,0
   host-23,6,0
   host-24,0,0
   host-25,0,0
   
   decomState 6 means the host is in vSAN Decom
   
   
3. If any hosts are in vSAN Decom follow KB vSAN Host Maintenance Mode is in sync with vSAN Node Decommission State to get the hosts out of vSAN Decom.
   
   
4. Ensure no hosts are in ESXi Maintenance Mode. If there are, take the hosts out of maintenance mode
   
   
5. Check vSAN Health Service for any failed disks/disk groups. If failed disks/disk groups exist, it is best to open a case with Broadcom Support for further assistance before replacing any disks, to assess the inaccessible objects and potentially avoid data loss. 
   
   
6. Check vSAN Health Service if there are any network partitions. If there are network partitions resolve any potential networking issues.
   
   
7. Check vSAN Health Service and verify the health for vSAN daemons clomd, epd, cmmdsd and vsanmgmtd.
   
   
8. See if one of the below KBs matches your issue.
   VMs with large-sized vmdks on vSAN ESA may become inaccessible
   Inaccessible objects in vSAN when all components show as Active
   Warning "Objects are in inaccessible in vSAN" while performing vSAN on-disk upgrade
   Cross-vCenter migration of VMs with snapshots to vSAN ESA may cause inaccessible objects during data resync.
   vSAN Skyline health reports an inaccessible object for a PVC with Read Write Many access mode using vSAN File Services

If none of the above clear the inaccessible objects then open a case with Broadcom Support for further assistance

Run the below commands and provide the collected info when you open the case.

esxcli vsan debug object health summary get
esxcli vsan debug object list --health=inaccessible > /tmp/inaccessible_objects.txt (Upload the file to the case once opened)