Tasks to Delete Persistent Volumes Keep Repeating and Failing

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

In vCenter's tasks, you frequently receive messages initiated by a vpxd-extension-####@vsphere.local account indicating: "Task Name: Delete a virtual storage object" or "Container volume deleting", or "Sync volume fails".

There is no impact to running workloads and new workloads can be deployed without issue.

You may see messages like this in the vsanvcmgmt.log:

2024-10-22T16:24:16.022Z info vsanvcmgmtd[3969721] [vSAN@6876 sub=Workflow opID=874ba70b] Successfully deleted disk ############-####-####-####-############

2024-10-22T16:24:37.706Z info vsanvcmgmtd[3969750] [vSAN@6876 sub=CnsSync] Starting sync ...
2024-10-22T16:24:37.706Z info vsanvcmgmtd[3969750] [vSAN@6876 sub=CnsSync] Sync all datastores ...
....
2024-10-22T16:24:40.391Z info vsanvcmgmtd[3969750] [vSAN@6876 sub=CnsSync] Process catalog change for FCD #########-####-####-####-############, vclock = ######, change = objectModified
2024-10-22T16:24:40.400Z info vsanvcmgmtd[3969750] [vSAN@6876 sub=CnsTask] A com.vmware.cns.vslm.tasks.retrieveVStorageObject task is created: task-46633612

 

Environment

vCenter Server 8.x

Resolution

The delete failure for the disks is due to the disk still being attached. Once the CSI detaches the volume, the volume deletes successfully. This is expected behavior.

In addition, some retrieveVStorageObjects triggered by CNS periodic sync are failing with NotFound because the volumes have been successfully deleted. These retrieveVStorageObject tasks will appear as failed in the vCenter UI and can be noisy. However, the NotFounds caused by the periodic sync can be ignored as they are expected.