How to identify the VM to which inaccessible pvc objects were attached.

Products

• VMware vSAN

Issue/Introduction

Symptoms: 

• Navigate to affected vSAN cluster > Monitor > Virtual objects. Any inaccessible pvc objects will list here. 
  

Environment

VMware vSAN 8.x

 

Resolution

There can be two scenarios here:

1. The PVC has not yet been deleted from Kubernetes side
2. The PVC has already been deleted from Kubernetes side 

• The PVC has not yet been deleted from Kubernetes side:

• • Make a note of the PVCs which are to be deleted.
  • On the vSphere Client select the vSAN Cluster > Monitor > Cloud Native Storage > Container Volumes. From here filter out to find the PVC which is to be deleted.
  • Select the PVC and view the details under "kubernetes objects" and "Basics" section.
     
    • To identify the volume ID:
      • Take a backup/snapshot of the vCenter Server Appliance. If the vCenter is in linked mode, the offline snapshot of all the vCenters are required.
      • Connect to the vCenter database:
        /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
      • Get the volume_id:
        
        VCDB=# select volume_id, volume_name  from cns.volume_info where volume_name like '%pvc-##############################%';

Output looks like:
                      volume_id               |               volume_name
        --------------------------------------+------------------------------------------
 ###########################          | pvc-##############################

        Note: volume_name corresponds to the PVC name displayed in the vCenter GUI.

      • Engage Kubernetes vendor to identify the VM name using the above mentioned volume ID

 

• The PVC has already been deleted from Kubernetes side: 

• • The information of pvc may not show on vCenter under Container Volumes. The names of the PVCs that were deleted from the Kubernetes side are required.
  • Generate the list of unassociated vSAN object UUIDs and path by using RVC. Follow the KB till step 8 - Procedures for identifying Unassociated vSAN objects.
  • Note the paths from the above list which end with .vmdk
  • From the SSH of a host in the cluster, run the cat command for the above path and look for the parameter "ddb.fcd.name =". This parameter displays the PVC name. If this parameter is not seen, this vmdk does not belong to a PVC.
    
    
    • To identify the volume ID:
      • Take a backup/snapshot of the vCenter Server Appliance. If the vCenter is in linked mode, the offline snapshot of all the vCenters are required.
      • Connect to the vCenter database: 
        /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
      • Get the volume_id:
        VCDB=# select volume_id, volume_name  from cns.volume_info where volume_name like '%pvc-##############################%';

        Output looks like:
                      volume_id               |               volume_name
--------------------------------------+------------------------------------------
 ###########################          | pvc-##############################

      • Engage Kubernetes vendor to identify the VM name using the above mentioned volume ID

Additional Information

Deleting Released Kubernetes PVCs from vSAN datastore

Container Volumes Displayed as Inaccessible Virtual Objects in vSAN Cluster.