Failed to Register Virtual Machine VM Service with Supervisor After Restoring VM From Snapshot via vCenter

Products

• VCF Private AI Services
• VMware Cloud Foundation

Issue/Introduction

A customer restored a VM Service VM from a snapshot taken via the vCenter UI, resulting in inconsistent VM states across management planes.

The following symptoms are all present:

• On the Supervisor (kubectl): The VM CR's spec.powerState is PoweredOff.
• On the Supervisor (kubectl): The VM CR has the vmoperator.vmware.com/paused annotation set to "true" with reason VirtualMachineBackupPaused.
• On vCenter: The vCenter VM shows as powered on, with the message: "Failed to register Virtual Machine VM service with Supervisor".

Environment

VMware Cloud Foundation (VCF) 9.0.x 

Cause

Taking and restoring snapshots of VM Service VMs directly from the vCenter Server UI is an unsupported operation in VCF 9.0.x and earlier.

Due to architectural limitations in the vCenter privilege model (required to permit legitimate backup/restore operations via VADP- based backup solutions like Veeam), this operation cannot be strictly blocked from the vCenter UI.

When a snapshot is reverted via the vCenter UI, the vCenter VM's ExtraConfig is rolled back to an older state containing an earlier backupVersion timestamp. The Supervisor's vmoperator detects this as a restore and pauses reconciliation to prevent conflicting updates. This leaves the VM in a paused state, preventing it from powering on.

Note: In VCF 9.1+, consumer VM snapshots are supported via the VirtualMachineSnapshot custom resource on the Supervisor. This is the supported method to snapshot VM Service VMs.

Resolution

To resolve the issue, remove the vmoperator.vmware.com/paused annotation from the VM CR and set its spec.powerState to PoweredOn.

Connect to the Supervisor cluster context.

Remove the paused annotation (note the "-" at the end of the annotation name):

kubectl annotate vm -n <namespace> <vm-name> vmoperator.vmware.com/paused-

Verify that the annotation has been successfully removed:

kubectl get vm -n <namespace> <vm-name> -o=jsonpath={.metadata.annotations}

Set the VM's desired power state to PoweredOn:

kubectl patch vm <vm-name> -n <namespace> -p '{"spec":{"powerState":"PoweredOn"}}' --type=merge

Verify that the VM has powered on:

kubectl get vm -n <namespace> <vm-name>