VCD Primary Cell CPU Usage Spikes to 100%

Products

• VMware Cloud Director

Issue/Introduction

• The CPU usage of the VMware Cloud Director Primary Cell experiences a substantial increase, ultimately reaching 100%.
• Restarting the VCD services provides a temporary reduction in CPU load, but the spike reoccurs shortly thereafter.

Environment

VMware Cloud Director 10.X

Cause

This issue occurs under the following circumstances:

• VCD services are stopped on the VCD Standby Cells, leaving the Primary VCD Cell to handle the full workload.
• The Load Balancer routes all traffic and requests to the Primary VCD Cell, leading to CPU saturation.

Resolution

• Verify if VCD services are stopped on the Standby VCD Cells. If so, investigate whether this is due to: Error :"All host(s) tried for query failed" when starting VMware Cloud Director services on the cell having Cassandra configured
• On the Primary VCD Cell, review the \opt\vmware\vcloud-director\logs\request.log,  file to identify IP addresses generating high request volumes. The log format is as follows:

<Source IP1> - - [05/Feb/2025:17:00:31 +0000] "HEAD /api/server_status HTTP/1.1" 503 18 "-" "vcd-watchdog" 2
<Source Ip2> - - [05/Feb/2025:17:01:39 +0000] "HEAD /api/server_status HTTP/1.1" 404 374 "-" "vcd-watchdog" 40<Source IP1>

• Validate the identified IP addresses to confirm if they include the Load Balancer IP.
• If confirmed, configure the Load Balancer to distribute load-balanced requests across all VCD Cells instead of directing them solely to the Primary VCD Cell.