CNS volume have disappeared from the CNS database

Products

• VMware vCenter Server

Issue/Introduction

PVC that is still attached to a worker node has accidentally been removed from the database.

Running kubectl get pvc -n <namespace> | grep <pvc name> still shows the volumes

Using Govc to list volumes or checking the database we can see the volume does not exist

select * from cns.volume_info where volume_name ~ 'pvc-<id>';

VCDB=# select * from cns.volume_info where volume_name ~ 'pvc-<>';
 volume_id | volume_name | volume_type | mark_for_delete | static_provision | datastore | vclock | is_cns_volume | disk_path | backing_object_id | capacity | last_modified_date | used_capacity | create_time | allocated_space
-----------+-------------+-------------+-----------------+------------------+-----------+--------+---------------+-----------+-------------------+----------+--------------------+---------------+-------------+-----------------
(0 rows)

 

From csi logs we can see below,

"level":"info","time":"2024-12-09T14:01:38.584751362Z","caller":"syncer/fullsync.go:776","msg":"FullSync for VC <VC FQDN>: create is required for volume: \"<volume-id>\""}

{"level":"warn","time":"2024-12-09T14:01:38.666600062Z","caller":"syncer/fullsync.go:460","msg":"FullSync for VC <VC FQDN>: Failed to create volume with the spec: (types.CnsVolumeCreateSpec) {\n DynamicData: (types.DynamicData) {\n },\n Name: (string) (len=40) \"pvc-<id>\",\n VolumeType: (string) (len=5) \"BLOCK\",\n Datastores: ([]types.ManagedObjectReference) <nil>,\n Metadata: ...   EntityType: (string) (len=23) \"PERSISTENT_VOLUME_CLAIM\",\n      EntityName: (string) (len=27) \"<pod-name>\",\n      Namespace: (string) (len=24) \"<namespace>\",\n      ClusterID: (string) (len=8) \"<cluster-id>\"\n     }\n    }\n   })\n  },\n  ContainerClusterArray: ([]types.CnsContainerCluster) (len=1 cap=1) {\n   (types.CnsContainerCluster) {\n    DynamicData: (types.DynamicData) {\n    },\n    ClusterType: (string) (len=10) \"KUBERNETES\",\n    ClusterId: (string) (len=8) \"<cluster-id>\",\n  BackingDiskUrlPath: (string) \"\",\n  BackingDiskObjectId: (string) \"\"\n }),\n Profile: ([]types.BaseVirtualMachineProfileSpec) <nil>,\n CreateSpec: (types.BaseCnsBaseCreateSpec) <nil>,\n VolumeSource: (types.BaseCnsVolumeSource) <nil>\n}\n. Err: ServerFaultCode: The object or item referred to could not be found."}

 

Environment

vSphere 7.0

vSphere 8.0

Cause

Accidental removal or a synchronization issue between datastore and database

Resolution

Resync the datastore in the vCenter database.

Connect to VCDB

psql -d VCDB -U postgres

Confirmed volume was missing.

select * from cns.volume_info where volume_name ~ 'pvc-<id>';

Resync of datastore, using MOB

https://<VCIP>/vslm/mob//?moid=StorageLifecycleManager&method=VslmSyncDatastore

 Datastore url is taken from VC UI and looks like ds:///vmfs/volumes/63c3eecf-cc6eb678-8d30-00**********.
  fullSync=true
  fcdId should be blank

Restart vsan-health service
     service-control --restart vsan-health

Connect to DB again and confirm volume is now showing .

select * from cns.volume_info where volume_name ~ 'pvc-<id>';

Additional Information

Should issue persist after attempting this, further troubleshooting may be needed.