CNS volume has disappeared from the CNS database

Products

• VMware vCenter Server
• Tanzu Kubernetes Runtime
• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• PVC that is still attached to a worker node has accidentally been removed from the database.
  
  
• Unable to resize/detach/attach pvc
  
  
• Running the following command still shows the volumes:
  kubectl get pvc -n <namespace> | grep <pvc name> 
  
  
• Validating the vCenter Server database it is seen that the volume does not exist:
  
  select * from cns.volume_info where volume_name ~ 'pvc-<id>';

VCDB=# select * from cns.volume_info where volume_name ~ 'pvc-<>';
 volume_id | volume_name | volume_type | mark_for_delete | static_provision | datastore | vclock | is_cns_volume | disk_path | backing_object_id | capacity | last_modified_date | used_capacity | create_time | allocated_space
-----------+-------------+-------------+-----------------+------------------+-----------+--------+---------------+-----------+-------------------+----------+--------------------+---------------+-------------+-----------------
(0 rows)
  
  
• Verifying the csi logs, following errors are seen:
  
  "level":"info","time":"YYYY-MM-DDTHH:MM:SSZ","caller":"syncer/fullsync.go:776","msg":"FullSync for VC <VC FQDN>: create is required for volume: \"<volume-id>\""}
{"level":"warn","time":"YYYY-MM-DDTHH:MM:SSZ","caller":"syncer/fullsync.go:460","msg":"FullSync for VC <VC FQDN>: Failed to create volume with the spec: (types.CnsVolumeCreateSpec) {\n DynamicData: (types.DynamicData) {\n },\n Name: (string) (len=40) \"pvc-<id>\",\n VolumeType: (string) (len=5) \"BLOCK\",\n Datastores: ([]types.ManagedObjectReference) <nil>,\n Metadata: ...   EntityType: (string) (len=23) \"PERSISTENT_VOLUME_CLAIM\",\n      EntityName: (string) (len=27) \"<pod-name>\",\n      Namespace: (string) (len=24) \"<namespace>\",\n      ClusterID: (string) (len=8) \"<cluster-id>\"\n     }\n    }\n   })\n  },\n  ContainerClusterArray: ([]types.CnsContainerCluster) (len=1 cap=1) {\n   (types.CnsContainerCluster) {\n    DynamicData: (types.DynamicData) {\n    },\n    ClusterType: (string) (len=10) \"KUBERNETES\",\n    ClusterId: (string) (len=8) \"<cluster-id>\",\n  BackingDiskUrlPath: (string) \"\",\n  BackingDiskObjectId: (string) \"\"\n }),\n Profile: ([]types.BaseVirtualMachineProfileSpec) <nil>,\n CreateSpec: (types.BaseCnsBaseCreateSpec) <nil>,\n VolumeSource: (types.BaseCnsVolumeSource) <nil>\n}\n. Err: ServerFaultCode: The object or item referred to could not be found."}

Environment

vSphere 7.0, 8.0
TKR 1.27.11
TKGM 2.5.1

Cause

Accidental removal or a synchronization issue between datastore and database

Resolution

1. Resync the datastore in the vCenter database.

   1. Connect to VCDB
      psql -d VCDB -U postgres


   2. Confirm that the volume is missing.
      select * from cns.volume_info where volume_name ~ 'pvc-<id>';


   3. Resync the datastore, using MOB
      https://<VCIP>/vslm/mob//?moid=StorageLifecycleManager&method=VslmSyncDatastore
      
      
   4.  Datastore url is taken from VC UI and looks like ds:///vmfs/volumes/########-########-####-############.
       fullSync=true
 fcdId should be blank


   5. Restart vsan-health service
      service-control --restart vsan-health


   6. Connect to DB again and confirm volume is now showing .
      select * from cns.volume_info where volume_name ~ 'pvc-<id>';
      
      
   7. If you find that the disk_path field references an incorrect disk or path.  You will want to force a sync inventory by Reconciling Discrepancies in the Managed Virtual Disk Catalog