Deployment stuck in 'WAITING FOR CONTROLLERS'

Products

• VMware Integrated OpenStack

Issue/Introduction

• The deployment status is stuck in 'WAITING FOR CONTROLLERS'
• osctl get viomachineset -o custom-columns=Name:.metadata.name,Count:.spec.count shows the desired count (e.g., 3), but the cluster status is not 'RUNNING'
• osctl get machine lists the expected number of machine objects.
• The corresponding Virtual Machine for one or more controllers is missing from the vCenter inventory or has been deleted/moved from the VIO resource pool.
• The management plane does not automatically trigger a redeployment of the missing node.

Environment

7.x

Cause

The issue is caused by an out-of-band deletion of a controller VM directly within vCenter.

In a Cluster API (CAPI) managed environment like VIO, the lifecycle of VMs is managed by controllers that track Custom Resources (CRs). When a VM is deleted via vCenter instead of via VIO/Kubernetes APIs, the Machine CR remains in the database with stale metadata. Because the Machine object still exists, the viomachineset controller believes the desired state is met and does not attempt to provision a replacement, leading to a state of desynchronization.

Resolution

To resolve the desynchronization and trigger a redeployment, the stale Machine object must be manually removed to allow the reconciliation loop to function.

1. Log in to the VIO Manager.
2. Identify the missing controller by comparing the Machine CRs against the vCenter inventory:
   #osctl get machine -o wide
3. Cross-reference the names from the output above with the VMs visible in the vCenter Resource Pool dedicated to VIO.
4. Delete the stale Machine CR for the VM that no longer exists in vCenter:
   #osctl delete machine <MACHINE_NAME>
5. Monitor the viomachineset to verify that a new machine is being provisioned:
   #watch osctl get viomachineset
6. Once the new VM is provisioned, customized, and joins the cluster, verify the deployment status:
   #viocli get deployment