VCF Management Services cluster or the VCF Automation cluster does not automatically recover after powering on VMs following a graceful shutdown

Products

• VCF Automation

Issue/Introduction

• VMs power on successfully but VCF Management Services UI (Fleet Lifecycle) remains inaccessible after 20+ minutes.
• VCF Automation UI shows components as unavailable or in error state.
• Services appear to be at 0 replicas and do not scale up automatically.
• Kubernetes nodes remain in Ready,SchedulingDisabled state indefinitely.

Environment

• VCF Management Services Runtime 9.1.0.x
• VCF Automation 9.1.0.x

Cause

During graceful shutdown, the internal cluster management service may mark certain nodes for deletion. After power-on, these nodes remain in a cordoned state, preventing the automatic recovery service from completing. The recovery process waits for all nodes to be usable before scaling services back up, creating a deadlock for bring up.

Resolution

Step 1: Power on the cluster VMs

1. Login to Management vCenter
2. Navigate to VMs and Templates view.
3. Locate the appropriate VM folder:
   • VCF Management Services cluster: vcf-management-services folder
   • VCF Automation cluster: vcf-automation folder
4. Identify the Control Plane VMs - Look for VMs with less CPU and Memory in the folder.
   • Note: For Automation all nodes are Control and Worker nodes simultaneously. 
5. Select all Control Plane VMs in the cluster.
   • Right-click and select Power → Power On.
   • Once powered on, wait for 5 minutes.
6. Select the Worker Node VMs.
   • Right-click and select Power → Power On.

 

Step 2: Wait for automatic recovery

Allow 15-20 minutes for the automatic recovery process to complete. The platform includes a systemd service that automatically scales services back to their original replica counts.

 

Step 3: Validate cluster recovery

Verify the cluster is operational by checking:

1. UI Access: Navigate to Fleet Lifecycle Manager UI (for management clusters) or VCF Automation services UI (for automation clusters). The UI should be accessible and responsive
2. Service Status: If UI access is unavailable, engage Broadcom Technical Support.

 

Step 4: Troubleshoot if automatic recovery fails

If services do not recover automatically after 20 minutes, manual intervention is required.

Access the cluster using ssh and the "vmware-system-user" user account.

1. Identify a control plane node IP address from vCenter (see above)
2. SSH to the control plane node as vmware-system-user
3. Switch to root (enter the "vmware-system-user" password): 
sudo -i
4. Set kubeconfig: 
   export KUBECONFIG=/etc/kubernetes/admin.conf
5. Validate pod status and services status
   •  Check node status - look for nodes stuck in "Ready,SchedulingDisabled" state:
     kubectl get nodes
   • Check if power-off-marker still exists (indicates recovery not completed):
     kubectl get configmap power-off-marker -n vmsp-platform
   • Check pod status for status that is not "Completed" or "Running" across all namespaces (Will return empty list if all pods are either completed or running):
     kubectl get pods -A | grep -v "Completed\|Running"
     
     

Manual recovery if in bad state

If nodes are stuck in "Ready,SchedulingDisabled" state and the power-off-marker "ConfigMap" exists, then you will need to run the following script on the control node that you are logged in for manual recovery:

1. Upload the script attached to the KB
2. Switch to root (enter the "vmware-system-user" password): 
sudo -i
3. Set kubeconfig: 
   export KUBECONFIG=/etc/kubernetes/admin.conf
4. Change to directory that script is uploaded and make it executable:
   chmod +x cluster-manual-recovery.sh
5. Run the script:
   ./cluster-manual-recovery.sh

In some race cases, the power-off-marker configMap could be deleted prematurely. If this is the case, above script can not be used and we need to perform the remediate steps in the script manually.

1. Uncordon non-condemned nodes
2. Scale VMSP platform and tenant services. Since the scaling data is stored in power-off-marker and is absent, these following things can be tried to retrieve scaling data: 
   1. Node journal from earlier recoveries (most reliable). On each management VM run: 
       
      journalctl -u service-startup.service --no-pager | grep -E "Scaling up (StatefulSet|Deployment)|Scaling .* to [0-9]+ replicas"

       Use the entries from the last recovery that restored a healthy marker, not the final one, which restored zeros.  Check journalctl --list-boots.
      
      
   2. Earlier shutdown workflow logs. The shutdown script logs the snapshot it wrote or the marker it reused (ConfigMap content found: ...).
      Pull these from the oldest system-shutdown workflow whose values are non-zero. The oldest is more likely to show the healthy state. 
      
      for wf in $(kubectl get workflows -n vmsp-platform --sort-by=.metadata.creationTimestamp -o name | grep -i shutdown); do   echo "=== ${wf}";   kubectl logs -n vmsp-platform -l workflows.argoproj.io/workflow="${wf#*/}" --all-containers --tail=-1 2>/dev/null     | grep -E "ConfigMap content found|MARKER TENANT_DATA|TENANT_DATA:|VMSP MARKER DATA|Entry data"; done
      
      
   3. PVC ordinals for StatefulSets. PVCs survive scale-down (default retention is Retain). The replica count is the highest PVC ordinal plus one (<template>-<sts>-0..N-1). Treat it as an upper bound, since an earlier scale-down can leave extra PVCs.
3. Verify recovery status

 

 

Step 5: Validate final state

After manual recovery, verify:

1. There should not be pods status in status that is not "Completed" or "Running" across all namespaces (Will return empty list if all pods are either completed or running): 
   kubectl get pods -A | grep -v "Completed\|Running"

2. Fleet Lifecycle Manager UI is accessible (for management clusters)

3. VCF Automation services are accessible (for automation clusters)