Recurring Persistent Volume Deletion and Sync Task Failures
search cancel

Recurring Persistent Volume Deletion and Sync Task Failures

book

Article ID: 381815

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service VMware vCenter Server

Issue/Introduction

Administrators may observe a high volume of recurring, failed tasks in the vCenter Server UI. These tasks are initiated by the Cloud Native Storage (CNS) service account (vpxd-extension-####@vsphere.local).

Specifically, you will see the following task names fail repeatedly:

  • Delete a virtual storage object

  • Container volume deleting

  • Sync volume fails

Key Observations:

  • No Operational Impact: Running workloads are unaffected, and new persistent workloads can be provisioned and deployed without issue.

  • Log Signatures: Reviewing the vsanvcmgmt.log reveals that the disks are eventually deleted successfully, but are followed by periodic CNS sync actions generating retrieveVStorageObject tasks.

Example vsanvcmgmt.log Excerpt:

YYYY-MM-DDTHH:MM:SS.022Z info vsanvcmgmtd[3969721] [vSAN@6876 sub=Workflow opID=874ba70b] Successfully deleted disk ############-####-####-####-############
YYYY-MM-DDTHH:MM:SS.706Z info vsanvcmgmtd[3969750] [vSAN@6876 sub=CnsSync] Starting sync ...
YYYY-MM-DDTHH:MM:SS.706Z info vsanvcmgmtd[3969750] [vSAN@6876 sub=CnsSync] Sync all datastores ...
....
YYYY-MM-DDTHH:MM:SS.391Z info vsanvcmgmtd[3969750] [vSAN@6876 sub=CnsSync] Process catalog change for FCD #########-####-####-####-############, vclock = ######, change = objectModified
YYYY-MM-DDTHH:MM:SS.400Z info vsanvcmgmtd[3969750] [vSAN@6876 sub=CnsTask] A com.vmware.cns.vslm.tasks.retrieveVStorageObject task is created: task-46633###

Environment

VMware vCenter Server 8.x

Cause

This is cosmetic, expected behavior caused by two overlapping mechanisms in the Container Storage Interface (CSI) and Cloud Native Storage (CNS) workflows:

  1. CSI Detach Latency: Initial attempts to delete a First Class Disk (FCD) will fail if the volume is still mounted/attached to a worker node. The deletion task will keep repeating until the CSI driver successfully completes the detach operation. Once detached, the volume deletes successfully.

  2. CNS Periodic Sync Race Condition: CNS relies on a periodic sync mechanism to align the vCenter storage catalog with the datastore. During this sync, CNS attempts to process a catalog change for the recently deleted disk. It triggers a retrieveVStorageObject API call to query the disk's status. Because the disk was just successfully deleted, the query returns a NotFound exception. This exception bubbles up to the vCenter UI as a failed task, creating administrative noise.

Resolution

No action is required. Because this is an expected byproduct of the asynchronous CSI detach process and CNS catalog synchronization, these task failures are benign. The transient NotFound errors and initial deletion failures can be safely ignored, as the system is functioning as designed and correctly cleaning up the persistent volumes in the background.

Powered by Wolken Software