"CPU usage is high", Control plane nodes have error on the vCenter UI

Products

• VMware vSphere with Tanzu

Issue/Introduction

• Guest cluster control plane nodes are crashing and report an error on the vCenter UI "CPU usage is high"
• The console of the CP VM have error, "Out of memory"

   

•  The CP VMs do not have any IP address assigned 
•  Upon rebooting the CP VMs, the VMs temporarily acquire an IP address. However, once the out-of-memory (OOM) issue occurs, the IP address is lost, and SSH access to the CP VMs is no longer possible.
•  While the VMs lose their IP addresses, the wcpmachine object still retains the IP information.

 

Environment

vSphere with Tanzu 7.x

vSphere with Tanzu 8.x

Cause

The CP nodes do not have enough resources. 

 

Resolution

1. From Supervisor, modify the TKC yaml to update the vmClass of the CP nodes according to the resources required. 

2. If cluster is stuck in False state after updating the vmClass of the CP nodes 

• Scale down vmop deployment to 0
  
  
  • k scale deployment vmware-system-vmop-controller-manager -n vmware-system-vmop --replicas=0

• Do these steps on the Affected CP Node. Repeat the same steps if more than one CP node is affected with same out of memory issue
  
  
  • Login to the ESXi host of the CP VM
  • Select the CP VM
  • Click on Edit Settings
  • Modify the CPU and Memory as per the required vmClass 

• Scale up the vmop deployment
  
  
  • k scale deployment vmware-system-vmop-controller-manager -n vmware-system-vmop --replicas=<required_number>

• Once the first CP node is stable, the cluster will start reconciling for all the CP nodes.