PV attachment to a pod fails with the error: The object or item referred to could not be found
search cancel

PV attachment to a pod fails with the error: The object or item referred to could not be found

book

Article ID: 451180

calendar_today

Updated On:

Products

VMware Telco Cloud Automation VMware vCenter Server VMware Telco Cloud Platform

Issue/Introduction

When attempting to attach or detach a PersistentVolume (PV) to/from a Kubernetes worker node VM using the vSphere CSI driver, pods remain stuck in ContainerCreating or Terminating states.

The following warning events are observed in kubectl describe pod or kubectl get events:

Warning  FailedAttachVolume  attachdetach-controller  
AttachVolume.Attach failed for volume "pvc-uuid" : rpc error: code = Internal desc = failed to attach disk: "disk-uuid" with node: "node-uuid" err failed to attach cns volume: "disk-uuid" to node vm... 
fault: (*types.LocalizedMethodFault)... Fault: (*types.ResourceInUse) LocalizedMessage: "The resource 'volume' is in use."

Additionally, the vSphere CSI controller logs contain detach failures:

Error processing "csi-uuid": failed to detach: rpc error: code = Internal desc = failed to detach disk: "disk-uuid" from node: "node-uuid" err failed to detach cns volume... 
fault: (*types.LocalizedMethodFault)... Fault: (*types.NotFound) LocalizedMessage: "The object or item referred to could not be found."

Environment

TCA 3.2

TCP 5.0

vSphere 8.x

Cause

This issue occurs due to a desynchronization between the vSphere Cloud Native Storage (CNS) database and the actual virtual disk attachment state. The CNS metadata indicates the volume is still attached to a previous worker node (stale lock), preventing the vSphere CSI driver from reassigning the disk to the new target node.

Resolution

Step 1: Manually Detach via vSphere Client UI

  1. Log in to the vSphere Client.

  2. Locate the Worker Node VM where the volume was previously attached (refer to the node UUID in the NotFound log).

  3. Right-click the VM > Edit Settings.

  4. Check if the hard disk corresponding to the Persistent Volume ID (disk-uuid) is still attached to the VM.

  5. If present, click the X icon to remove the hard disk (do not delete files from the datastore).

  6. Click OK to save changes.

Step 2: Force Clean Up VolumeAttachment Objects in Kubernetes

  1. Identify the failing VolumeAttachment object in the Kubernetes cluster:

    kubectl get volumeattachment | grep <pvc-name-or-id>

  2. Delete the stuck VolumeAttachment resource:

    kubectl delete volumeattachment <volume-attachment-name>

  3. Restart the affected pod or trigger a force re-scheduling by scaling down and back up the deployment:

    kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi

Step 3: Resynchronize vCenter CNS Datastore

If the volume remains in an inconsistent state in the Cloud Native Storage cache, trigger a CNS resync operation via the vCenter API as outlined in Broadcom KB 383797.

Confirm the volume is successfully attached to the new node and the Pod status transitions to Running.

Powered by Wolken Software