'vSAN Cluster Configuration Consistency' due to CPU over-provisioning

Products

• VMware vSAN

Issue/Introduction

This article identifies a scenario where vSAN health alarms, including "Hosts with connectivity issues" and "vSAN Cluster Configuration Consistency," are triggered by severe CPU over-provisioning on ESXi hosts.

• vSAN Skyline Health reports "Hosts with connectivity issues" for one or more hosts.
• Cluster reports "vSAN Cluster Configuration Consistency" errors.
• Skyline health indicates "Hosts with connectivity issues" intermittently or persistently.
• High latency is observed across the vSAN cluster.
• ESXi hosts are connected to vCenter, but vSAN API calls fail or time out.

Environment

• VMware vSAN 8.0.x
• VMware ESXi 8.0U3
• High vCPU to pCPU over-provisioning ratios (e.g., ratios exceeding 150%).

Cause

Severe compute resource contention occurs when the vCPU to pCPU ratio reaches critical levels. This over-provisioning causes processing delays for the vSAN management services (such as vsanmgmtd), leading to failed API calls and inconsistent health reporting between the ESXi hosts and vCenter.

Resolution

To resolve this issue, reduce the compute resource contention across the affected cluster:

1. Evaluate Over-provisioning: Review log data or performance metrics to identify hosts where the vCPU to pCPU ratio is in the "Yellow" or "Red" zone (e.g., ratios of 160% to 240%).
2. Immediate Relief: Power off non-essential Virtual Machines (VMs) or migrate workloads to clusters with available capacity to lower the current CPU demand.
3. Right-size Workloads: Adjust the number of vCPUs assigned to VMs to more accurately reflect their actual utilization.
4. Capacity Expansion: If workload demand remains high, add additional compute nodes to the vSAN cluster to balance the vCPU load.

 

Additional Information

For Ref-

Host CPU overutilization may negatively impact vSAN performance