Deleting Released Kubernetes PVCs from vSAN datastore
searchcancel
Deleting Released Kubernetes PVCs from vSAN datastore
book
Article ID: 368545
calendar_today
Updated On: 07-24-2024
Products
VMware vSAN
Issue/Introduction
PVCs (Persistent Volume Claim) with reclaim policy of "Retain", if deleted from Kubernetes side, does not get deleted from the vSAN Storage.
The consumed space on the vSAN datastore does not decrease.
In the vSAN Capacity Overview tab, the deleted PVC's storage utilization shows up under User objects > Improved virtual disks > Other improved virtual disks.
The vSAN objects for the deleted PVC appear as unassociated on vSAN.
Environment
VMware vSAN 7.0.x
Cause
This is expected behavior for PVCs with reclaim policy "Retain" on the Kubernetes side.
With the "Retain" policy, if a user deletes a PersistentVolumeClaim, the corresponding PersistentVolume will not be deleted. Instead, it is moved to the Released phase, where all of its data can be manually recovered.
This policy is useful if the volume contains critical data.
Due to this, the storage space is not released from the vSAN datastore.
Resolution
If the PVCs need to be permanently deleted and its space utilization is to be released, follow the steps below:
WARNING: THIS OPERATION IS IRREVERSIBLE. WARNING: DOUBLE-CHECK ALL PARAMETERS - PVC NAME, VMDK PATH, ddb.fcd.name.
There can be two scenarios here:
The PVC has not yet been deleted from Kubernetes side
The PVC has already been deleted from Kubernetes side
1. The PVC has not yet been deleted from Kubernetes side:
Make a note of the PVCs which are to be deleted.
On the vSphere Client select the vSAN Cluster > Monitor > Cloud Native Storage > Container Volumes. From here filter out to find the PVC which is to be deleted.
Select the PVC and view its basic details. Here the Volume Path will be listed. Make a note of it.
Now proceed to delete the PVC from Kubernetes side.
Once done, browse the datastore path for the PVC's .vmdk file via the vSphere client datastore browser. Select the same .vmdk file as shown in the path (not the -ctk.vmdk file) and delete it. This should release the space from the vSAN datastore.
2. The PVC has already been deleted from Kubernetes side:
The names of the PVCs that were deleted from the Kubernetes side are required.
Note the paths from the above list which end with .vmdk
From the SSH of a host in the cluster, run the cat command for the above path and look for the parameter "ddb.fcd.name =". This parameter displays the PVC name. If this parameter is not seen, this vmdk does not belong to a PVC.
If it matches, browse the same path via vSphere Client datastore browser, select the same .vmdk file as shown in the path (not the -ctk.vmdk file) and delete it. This should release the space from the vSAN datastore.