Failed to delete VMs which are stuck at "Powering Off" state from VCF Automation UI

Products

• VCF Automation

Issue/Introduction

• Virtual Machines provisioned within VCF Automation (VCFA) remain stuck in a "Powering Off" state within the Tenant UI following Protection and Recovery replication workflows.
• In vCenter Server and raw resource YAML files, the affected VMs report as "Powered Off".
• Deletion requests initiated through the VCFA Tenant UI complete without returning errors, but the VM objects persist in the VCFA portal inventory.

Environment

VCF Automation 9.1

Cause

The VirtualMachine Custom Resource in the vSphere Supervisor namespace has the annotation [vmoperator.vmware.com/paused](https://vmoperator.vmware.com/paused): true applied, which explicitly halts VM Operator controller reconciliation loops and prevents state synchronization and deletion handling between vCenter and VCF Automation.

Resolution

To resolve the issue:

• Connect and authenticate to the vSphere Supervisor Cluster.
  

• Confirm the presence of the "paused" annotation on the stuck VirtualMachine custom resource:
  • kubectl describe vm <VM_NAME> -n <NAMESPACE_NAME> | grep -ie 'paused'
    • If the paused annotation is present, the expected output would be: vmoperator.vmware.com/paused: true
      
      
• Confirm if the VM is already marked for deletion:
  • kubectl describe vm <VM_NAME> -n <NAMESPACE_NAME> | grep -ie 'deletion'
    • If the delete operation on the target VM is already initiated, the expected outcome would be: 
      

      Deletion Grace Period Seconds: 0
Deletion Timestamp:            YYY-MM-DDT##:##:##z

• Remove the [vmoperator.vmware.com/paused](https://vmoperator.vmware.com/paused) annotation to resume VM Operator controller reconciliation:
  • kubectl annotate vm <VM_NAME> -n <NAMESPACE_NAME> vmoperator.vmware.com/paused
    • Output of the above command is: virtualmachine. vmoperator. ator.vnware.com/<VM_NAME> annotated
      
      
• Verify that the target VM resource finishes reconciliation and is removed from the Supervisor namespace: kubectl get vm <VM_NAME> -n <NAMESPACE_NAME>
  
  
• Navigate back to the VCF Automation Tenant UI and confirm the ghost VM object is successfully removed.