High CPU utilization on VCF Automation node due to policy-insights-server pods

Products

• VCF Automation

Issue/Introduction

• One or more VCF Automation (VCFA) nodes experience unusually high CPU utilization (e.g., consuming 70+ GHz), potentially choking the ESXi host where the VM resides.

• Running the command kubectl top pods -n prelude --sort-by=cpu on the affected node identifies the policy-insights-server pods (e.g., policy-insights-server-##### and policy-insights-server-#####) as the primary source of the high CPU usage.

• Reviewing the logs for the policy-insights-server pod reveals Customer Experience Improvement Program (CEIP) errors indicating that the appliance is repeatedly failing to send telemetry data. Example log entries:

  [core] [Channel #3 SubChannel #4]grpc: addrConn. createTransport failed to connect to {Addr: "policy-engine-grpc: 443", ServerName: "policy-engine-grpc:443", }. Err: connection error: desc = "transport: Error while dialing: dial tcp IP-Address:443: connect: connection refused" subcomponent=grpc-runtime
  {"component": "policy-engine-bi", "level":"info", "msg":"CEIP data collection enabled; telemetry config vksm-vcfa.9_0, scURL: VMware-CEIP-Address every 24h0m0s", "time": "Time-Stamp"}

Environment

VCF Automation 9.0.x

Cause

This issue occurs because the policy-insights-server pod continuously attempts to send CEIP telemetry data but fails to establish a connection. The repetitive connection retry loop heavily consumes CPU resources on the VCF Automation appliance.

Resolution

To resolve the high CPU utilization, you must disable CEIP data collection and reboot the environment.

1. Disable CEIP data collection on the Aria Automation environment. Refer https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/provider-management/managing-system-settings/activate-or-deactivate-your-customer-experience-improvement-program-participation.html  for steps.

2. Reboot the Aria Automation instance to ensure the fix persists. Refer https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/fleet-management/vcf-shutdown-and-startup/sddc-startup/start-the-management-domain/start-the-vrealize-automation-cluster.html for steps.