VCF Management Services cluster or the VCF Automation cluster does not automatically recover after powering on VMs following a graceful shutdown

Products

• VCF Automation

Issue/Introduction

• VMs power on successfully but VCF Management Services UI (Fleet Lifecycle) remains inaccessible after 20+ minutes.
• VCF Automation UI shows components as unavailable or in error state.
• Services appear to be at 0 replicas and do not scale up automatically.
• Kubernetes nodes remain in Ready,SchedulingDisabled state indefinitely.

Environment

• VCF Management Services Runtime 9.1.0.x
• VCF Automation 9.1.0.x

Cause

During graceful shutdown, the internal cluster management service may mark certain nodes for deletion. After power-on, these nodes remain in a cordoned state, preventing the automatic recovery service from completing. The recovery process waits for all nodes to be usable before scaling services back up, creating a deadlock for bring up.

Resolution

Step 1: Power on the cluster VMs

1. Login to Management vCenter
2. Navigate to VMs and Templates view.
3. Locate the appropriate VM folder:
   • VCF Management Services cluster: vcf-management-services folder
   • VCF Automation cluster: vcf-automation folder
4. Identify the Control Plane VMs - Look for VMs with less CPU and Memory in the folder.
   • Note: For Automation all nodes are Control and Worker nodes simultaneously. 
5. Select all Control Plane VMs in the cluster.
   • Right-click and select Power → Power On.
   • Once powered on, wait for 5 minutes.
6. Select the Worker Node VMs.
   • Right-click and select Power → Power On.

 

Step 2: Wait for automatic recovery

Allow 15-20 minutes for the automatic recovery process to complete. The platform includes a systemd service that automatically scales services back to their original replica counts.

 

Step 3: Validate cluster recovery

Verify the cluster is operational by checking:

1. UI Access: Navigate to Fleet Lifecycle Manager UI (for management clusters) or VCF Automation services UI (for automation clusters). The UI should be accessible and responsive
2. Service Status: If UI access is unavailable, engage Broadcom Technical Support.

 

Step 4: Troubleshoot if automatic recovery fails

If services do not recover automatically after 20 minutes, manual intervention is required.

Access the cluster using ssh and the "vmware-system-user" user account.

1. Identify a control plane node IP address from vCenter (see above)
2. SSH to the control plane node as vmware-system-user
3. Switch to root (enter the "vmware-system-user" password): 
sudo -i
4. Set kubeconfig: 
   export KUBECONFIG=/etc/kubernetes/admin.conf
5. Validate pod status and services status
   •  Check node status - look for nodes stuck in "Ready,SchedulingDisabled" state:
     kubectl get nodes
   • Check if power-off-marker still exists (indicates recovery not completed):
     kubectl get configmap power-off-marker -n vmsp-platform
   • Check pod status for status that is not "Completed" or "Running" across all namespaces (Will return empty list if all pods are either completed or running):
     kubectl get pods -A | grep -v "Completed\|Running"
     
     

Manual recovery if in bad state

If nodes are stuck in "Ready,SchedulingDisabled" state and the power-off-marker "ConfigMap" exists, then you will need to run the following script on the control node that you are logged in for manual recovery.

1. Upload the script attached to the KB
2. Switch to root (enter the "vmware-system-user" password): 
sudo -i
3. Set kubeconfig: 
   export KUBECONFIG=/etc/kubernetes/admin.conf
4. Change to directory that script is uploaded and make it executable:
   chmod +x cluster-manual-recovery.sh
5. Run the script:
   ./cluster-manual-recovery.sh

 

Step 5: Validate final state

After manual recovery, verify:

1. There should not be pods status in status that is not "Completed" or "Running" across all namespaces (Will return empty list if all pods are either completed or running): 
   kubectl get pods -A | grep -v "Completed\|Running"

2. Fleet Lifecycle Manager UI is accessible (for management clusters)

3. VCF Automation services are accessible (for automation clusters)