SSP Worker Nodes Get Stuck in Provisioning State Due to Insufficient CPU Resources

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

After completing a fresh installation of the Security Services Platform (SSP) platform and worker nodes, all nodes respond to ping requests. However, the following issues occur:

• Pod Scheduling Failure: One or more pods fail to schedule, throwing an Insufficient cpu error (e.g., FailedScheduling event).
  
  
• Management UI Unreachable: The management web UI fails to load or displays a browser connection error.
  
  
• Worker Node Provisioning Failure: If a worker node is powered off via vCenter and its vCPU allocation is increased (e.g., from 16 to 32 vCPUs), powering it back on results in the node remaining stuck in a Provisioning state.

Environment

SSP 5.0 and above

Cause

The underlying ESXi hypervisor cluster lacks sufficient physical CPU capacity to fulfill the resource requests (such as increasing vCPU allocation on worker nodes).

As a result, vSphere Admission Control prevents the VM from booting properly or scaling up, leaving Kubernetes pods in a persistent Pending state with FailedScheduling errors.

Resolution

1. Inspect Cluster Resources: Check the current CPU utilization and available capacity in the vSphere cluster/resource pool hosting the SSP nodes as per the system requirement of SSP.
   
   SSP System requirements
2. Confirm CPU Exhaustion: Verify if physical CPU host capacity is fully consumed across the underlying vSphere cluster.
   
   
3. Expand Cluster Capacity: Add additional physical CPU capacity to the vSphere cluster (e.g., by joining additional ESXi hosts to the cluster).
   
   
4. Scale / Power On SSP Nodes: Once total physical cluster capacity is expanded, power on or scale up the SSP worker node VMs.