Guest Cluster (TKC) worker nodes are getting rebuilt when memory usage goes more than 90% due to slab memory issue

Products

• VMware vSphere with Tanzu
• vSphere with Tanzu

Issue/Introduction

• Tanzu Kubernetes Guest Cluster worker node (Photon OS) reaches high memory usage, and the worker nodes gets rebuilt.
  
  
• Within the Guest Cluster worker nodes, the memory usage goes more than 90%, more than half of the memory is consumed by slab memory with TCP high usage.
  
  

Resolution

• To check slab memory details, you can type the below command within the worker node
  
  cat /proc/meminfo
cat /proc/sys/net/ipv4/tcp_max_orphans
slabtop --once --sort c | head -20
slabtop -s -a
grep TCP /proc/slabinfo 
  
  
• To verify the kernel-version for the Guest Cluster, you can run the below command: (RPM should be linux-4.19.290 or above)

kubectl get node -o wide | grep <guest-cluster-name>

 

To resolve this issue, you can upgrade the TKC Guest cluster to v1.26.13 or latest. 

 

Additional Information

https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-supervisor/8-0/release-notes/vmware-tkrs-release-notes.html