If a volume is removed from the VM specification while the volume attachment operation is still in progress, the system enters an inconsistent state

Products

• VMware vCenter Server

Issue/Introduction

This issue occurs during VM creation workflows when volumes are being attached to VMs. If a volume is removed from the VM specification while the volume attachment operation is still in progress (before the Node VM Attachment CR (Custom Resource) status has been updated), the system enters an inconsistent state with the following symptoms:

Symptoms (one or more may be present):

•  Volume remains attached to the VM in vCenter even after being removed from the VM specification
•  Node VM Attachment Custom Resource (CnsNodeVmAttachment CR) is not cleaned up and becomes stale
•  Subsequent VM creation with same volume attachment request fails with error indicating that volume is already attached to another VM

Environment

VC 8.0.3 P08 or earlier

Cause

This issue is caused by a race condition in the volume attachment workflow. When a volume attachment operation is initiated during VM creation, there is a window of time before the Node VM Attachment CR status is updated to reflect the in-progress operation. If the volume is removed from the VM specification during this window (either through user action or automated system decision), the attachment workflow does not properly reconcile the state. As a result:

1. The volume attachment completes on vCenter despite the volume being removed from the spec

2. The Node VM Attachment CR is not cleaned up

Resolution

Immediate Workaround: User needs to use the following manual steps to recover from this state:

1. Identify the affected volume and VM:

Check the VM specification to confirm which volume was removed 

Verify the volume is still attached in vCenter by examining the VM's storage configuration 

Identify the stale Node VM Attachment CR using: kubectl get cnsnodevmattachment -A


2. Manually detach the volume from the VM: 

In vCenter, navigate to the affected VM 

Edit VM settings and remove the attached volume

Confirm the detachment operation completes successfully


3. Clean up the stale Node VM Attachment CR:
   kubectl delete cnsnodevmattachment <cr-name> -n 
 <namespace>

If the CR is stuck in terminating state, remove the finalizer:
   kubectl patch cnsnodevmattachment <cr-name> -n 
 <namespace>  -p '{"metadata":{"finalizers":null}}' --type=merge



4. Verify cleanup:

Confirm the volume is no longer attached to any VM in vCenter

Confirm the Node VM Attachment CR is deleted

Retry the VM creation or volume attachment operation