PVC that is still attached to a worker node has accidentally been removed from the database.
Running kubectl get pvc -n <namespace> | grep <pvc name>
still shows the volumes
Using Govc to list volumes or checking the database we can see the volume does not exist
select * from cns.volume_info where volume_name ~ 'pvc-<id>';
VCDB=# select * from cns.volume_info where volume_name ~ 'pvc-<>';
volume_id | volume_name | volume_type | mark_for_delete | static_provision | datastore | vclock | is_cns_volume | disk_path | backing_object_id | capacity | last_modified_date | used_capacity | create_time | allocated_space
-----------+-------------+-------------+-----------------+------------------+-----------+--------+---------------+-----------+-------------------+----------+--------------------+---------------+-------------+-----------------
(0 rows)
From csi logs we can see below,
"level":"info","time":"2024-12-09T14:01:38.584751362Z","caller":"syncer/fullsync.go:776","msg":"FullSync for VC <VC FQDN>: create is required for volume: \"<volume-id>\""}
{"level":"warn","time":"2024-12-09T14:01:38.666600062Z","caller":"syncer/fullsync.go:460","msg":"FullSync for VC <VC FQDN>: Failed to create volume with the spec: (types.CnsVolumeCreateSpec) {\n DynamicData: (types.DynamicData) {\n },\n Name: (string) (len=40) \"pvc-<id>\",\n VolumeType: (string) (len=5) \"BLOCK\",\n Datastores: ([]types.ManagedObjectReference) <nil>,\n Metadata: ... EntityType: (string) (len=23) \"PERSISTENT_VOLUME_CLAIM\",\n EntityName: (string) (len=27) \"<pod-name>\",\n Namespace: (string) (len=24) \"<namespace>\",\n ClusterID: (string) (len=8) \"
<cluster-id>
\"\n }\n }\n })\n },\n ContainerClusterArray: ([]types.CnsContainerCluster) (len=1 cap=1) {\n (types.CnsContainerCluster) {\n DynamicData: (types.DynamicData) {\n },\n ClusterType: (string) (len=10) \"KUBERNETES\",\n ClusterId: (string) (len=8) \"<cluster-id>
\",\n BackingDiskUrlPath: (string) \"\",\n BackingDiskObjectId: (string) \"\"\n }),\n Profile: ([]types.BaseVirtualMachineProfileSpec) <nil>,\n CreateSpec: (types.BaseCnsBaseCreateSpec) <nil>,\n VolumeSource: (types.BaseCnsVolumeSource) <nil>\n}\n. Err: ServerFaultCode: The object or item referred to could not be found."}
vSphere 7.0
vSphere 8.0
Accidental removal or a synchronization issue between datastore and database
Resync the datastore in the vCenter database.
Connect to VCDB
psql -d VCDB -U postgres
Confirmed volume was missing.
select * from cns.volume_info where volume_name ~ 'pvc-<id>';
Resync of datastore, using MOB
https://<VCIP>/vslm/mob//?moid=StorageLifecycleManager&method=VslmSyncDatastore
Datastore url is taken from VC UI and looks like ds:///vmfs/volumes/63c3eecf-cc6eb678-8d30-00**********
.
fullSync=true
fcdId should be blank
Restart vsan-health service service-control --restart vsan-health
Connect to DB again and confirm volume is now showing .
select * from cns.volume_info where volume_name ~ 'pvc-<id>';
Should issue persist after attempting this, further troubleshooting may be needed.