VCF Management Services cluster or the VCF Automation cluster does not automatically recover after powering on VMs following a graceful shutdown

Products

• VCF Automation

Issue/Introduction

• VMs power on successfully but VCF Management Services UI (Fleet Lifecycle) remains inaccessible after 20+ minutes.
• VCF Automation UI shows components as unavailable or in error state.
• Services appear to be at 0 replicas and do not scale up automatically.
• Kubernetes nodes remain in Ready,SchedulingDisabled state indefinitely.

Environment

• VCF Management Services Runtime 9.1.0.0
• VCF Automation 9.1.0.0

Cause

During graceful shutdown, the internal cluster management service may mark certain nodes for deletion. After power-on, these nodes remain in a cordoned state, preventing the automatic recovery service from completing. The recovery process waits for all nodes to be usable before scaling services back up, creating a deadlock for bring up.

Resolution

Step 1: Power on the cluster VMs

Note: For Automation, skip to step 5, all Node VMs are Control and Worker, order does not matter.

1. Open VCF Operations UI and navigate to Build > Lifecycle > Components tab.
2. Click on the VCF Services Runtime link.
3. Scroll down the page to the Nodes section.
4. Identify the Control Plane nodes.
5. In vCenter, navigate to VMs and Templates view.
6. Locate the appropriate VM folder:
   • VCF Management Services cluster: vcf-management-services folder
   • VCF Automation cluster: vcf-automation folder
7. Select all Control Plane VMs in the cluster.
8. Right-click and select Power → Power On.
9. Once powered on, select the Worker Node VMs and power those on.

 

Step 2: Wait for automatic recovery

Allow 15-20 minutes for the automatic recovery process to complete. The platform includes a systemd service that automatically scales services back to their original replica counts.

 

Step 3: Validate cluster recovery

Verify the cluster is operational by checking:

1. UI Access: Navigate to Fleet Lifecycle Manager UI (for management clusters) or VCF Automation services UI (for automation clusters). The UI should be accessible and responsive
2. Service Status: If UI access is unavailable, engage Broadcom Technical Support.

 

Step 4: Troubleshoot if automatic recovery fails

If services do not recover automatically after 20 minutes, manual intervention is required.

Access the cluster using breakglass

1. Identify a control plane node IP address from vCenter
   
   
2. SSH to the control plane node as vmware-system-user
   
   

3. Login with breakglass password

4. Switch to root: sudo -i
   
   
5. Set kubeconfig: export KUBECONFIG=/etc/kubernetes/admin.conf

Validate pod status and service status

# Check node status - look for nodes stuck in SchedulingDisabled state

kubectl get nodes

 

# Check if power-off-marker still exists (indicates recovery not completed)

kubectl get configmap power-off-marker -n vmsp-platform

 

# Check pod status across all namespaces

kubectl get pods -A | grep -v Running

 

Manual recovery if in bad state

If nodes are stuck in Ready,SchedulingDisabled state and the power-off-marker ConfigMap exists, then you will need to run the following script on that same node for manual recovery after setting the KUBECONFIG variable: 

"cluster-manual-recovery.sh"

 

Step 5: Validate final state

After manual recovery, verify:

• All pods show Running status: kubectl get pods -A

• Fleet Lifecycle Manager UI is accessible (for management clusters)

• VCF Automation services are accessible (for automation clusters)