High Memory usage on control plan Nodes due to excessive requests from Application Pods

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

• High memory usage is observed on control plan nodes
• we see below log entry in audit log:

• YYYY-MM-DDTHH:MM:SS.","stageTimestamp":"YYYY-MM-DDTHH:MM:SS","annotations":{"authorization.k8s.io/decision":"allow","
  authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"smb-csi-provisioner-binding\" of ClusterRole \"smb-externa
  l-provisioner-role\" to ServiceAccount \"csi-smb-controller-sa/csi-smb\""}}
  {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"xxxx","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/namespace-system/pods/pod_name","verb":"delete","user":{"username":"system:node:control_plan_node_name","groups":["system:nodes","system:authenticated"]},"sourceIPs":["10.x.x.x"],"userAgent":"kubelet/v1.28.15+vmware.7 (linux/amd64) kubernetes/12345","objectRef":{"resource":"pods","namespace":"name_space","name":"pod_name","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"validate.pod.svc-fail\": failed to call webhook: Post \"https://example.com:443/validate/fail?timeout=10s\": dial tcp 10.x.x.x:443: connect: connection refused","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"validate.pod-svc-fail\": failed to call webhook: Post \"https://example.com:443/validate/fail?timeout=10s\": dial tcp 10.x.x.11:443: connect: connection refused"}]},"code":500},
  

Environment

TKGm 2.x

Cause

The volume of TCP request generated by the application pods exceeds the system RAM resource.

Resolution

The issue originates in the 3rd-party application logic so resolution requires an application-level patch or configuration change. A support request should be raised with the Application Vendor (CNF provider).