First Class Disks report as 'Inaccessible' under 'Cluster > Monitor > vSAN > Virtual Objects' while 'Cluster > Monitor > vSAN > Skyline Health' does not alert on data unavailability

Products

• VMware vCenter Server

Issue/Introduction

In some situations First Class Disks used for storage nodes by Kubernetes may report as 'Inaccessible' (under 'Cluster > Monitor > vSAN > Virtual Objects') - entries seen consist of volume name followed by volume_id.

This article is applicable when:

• Kubernetes operator for storage & data management (Portworx, Ceph, OpenShift, etc.) does not report any missing volumes,
• Skyline Health does not report inaccessible vSAN objects,
• VMDK file previously associated with PVC/PV volume does not exist on vSAN datastore.  

Environment

VMware vCenter Server 8.x

VMware vCenter Server 9.x

VMware vSphere Kubernetes Service

Cause

Deleting volume objects (PVC/PV VMDK descriptor files) directly from the vSAN datastore can lead to the situation when vCenter shows volumes associated with removed VMDK files as 'Inaccessible'. 

Resolution

To remove First Class Disks showing as 'Inaccessible' under 'Cluster > Monitor > vSAN > Virtual Objects', perform the following steps:
 
NOTE: Before making any changes, take a backup of the vCenter Server Appliance.
NOTE: Verify with the internal Kubernetes administration team that no Container Volumes are missing. 
 

1. Make a note of all 'Inaccessible' volume names seen under 'Cluster > Monitor > vSAN > Virtual Objects'
2. Log into vCenter Appliance as root using SSH 
3. Stop the vpxd service: 
   
   service-control --stop vpxd 
   
   
4. Connect to the vCenter database:
   
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
   
   
5. Obtain volume_id for each volume_name noted earlier and make a note of each volume_id <-> volume_name pair, example:
   

   VCDB=# select volume_id, volume_name from cns.volume_info where volume_name like '%PX-##############################%';

   
                 volume_id               |               volume_name
--------------------------------------+------------------------------------------
############################          | PX-##############################

   NOTE: volume_name corresponds to the PVC name displayed in the vCenter GUI.

6. Remove each entry using volume_id's noted in previous step, example:
   
   VCDB=# delete from cns.volume_info where volume_id='volume_id_from_step_5';
   
   
7. Type \q (or quit) and then press ENTER to quit psql shell.
8. Start the vpxd service:
   
   service-control --start vpxd
   
   
9. Refresh VCSA's UI and verify that no more inaccessible PVC volumes are shown under 'Cluster > Monitor > vSAN > Virtual Objects' 

Additional Information

Situation initially encountered with Portworx being used to manage First Class Disks.

No snapshots linked to the PVC volumes were found in this scenario; therefore, the approach outlined in the article Container Volume snapshot objects displayed as Inaccessible Virtual Objects in vSAN Cluster is not viable.