Deleting Released Kubernetes PVCs from vSAN datastore
search cancel

Deleting Released Kubernetes PVCs from vSAN datastore

book

Article ID: 368545

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

PVCs (Persistent Volume Claim) with reclaim policy of "Retain", if deleted from Kubernetes side, does not get deleted from the vSAN Storage.

The consumed space on the vSAN datastore does not decrease.

In the vSAN Capacity Overview tab, the deleted PVC's storage utilization shows up under User objects > Improved virtual disks > Other improved virtual disks.

The vSAN objects for the deleted PVC appear as unassociated on vSAN.

Environment

VMware vSAN 7.0.x

Cause

This is expected behavior for PVCs with reclaim policy "Retain" on the Kubernetes side.

With the "Retain" policy, if a user deletes a PersistentVolumeClaim, the corresponding PersistentVolume will not be deleted. Instead, it is moved to the Released phase, where all of its data can be manually recovered.

This policy is useful if the volume contains critical data.

Due to this, the storage space is not released from the vSAN datastore.

Resolution

If the PVCs need to be permanently deleted and its space utilization is to be released, follow the steps below:

WARNING: THIS OPERATION IS IRREVERSIBLE.
WARNING: DOUBLE-CHECK ALL PARAMETERS - PVC NAME, VMDK PATH, ddb.fcd.name.

 

There can be two scenarios here:

  1. The PVC has not yet been deleted from Kubernetes side
  2. The PVC has already been deleted from Kubernetes side

 

1. The PVC has not yet been deleted from Kubernetes side:

  1. Make a note of the PVCs which are to be deleted.
  2. On the vSphere Client select the vSAN Cluster > Monitor > Cloud Native Storage > Container Volumes. From here filter out to find the PVC which is to be deleted.
  3. Select the PVC and view its basic details. Here the Volume Path will be listed. Make a note of it.
  4. Now proceed to delete the PVC from Kubernetes side.
  5. Once done, browse the datastore path for the PVC's .vmdk file via the vSphere client datastore browser. Select the same .vmdk file as shown in the path (not the -ctk.vmdk file) and delete it. This should release the space from the vSAN datastore.

2. The PVC has already been deleted from Kubernetes side: 

  1. The names of the PVCs that were deleted from the Kubernetes side are required.
  2. Generate the list of unassociated vSAN object UUIDs and path by using RVC. Follow the KB till step 8 - Procedures for identifying Unassociated vSAN objects.
  3. Note the paths from the above list which end with .vmdk
  4. From the SSH of a host in the cluster, run the cat command for the above path and look for the parameter "ddb.fcd.name =". This parameter displays the PVC name. If this parameter is not seen, this vmdk does not belong to a PVC.
  5. If it matches, browse the same path via vSphere Client datastore browser, select the same .vmdk file as shown in the path (not the -ctk.vmdk file) and delete it. This should release the space from the vSAN datastore.

Additional Information

Powered by Wolken Software