VM performance degradation due to BIOS MM Config Base settings on Lenovo SR650 V3.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

Virtual Machines (VMs) across a cluster experience consistent performance degradation and slowness following an hardware upgrade to Lenovo ThinkSystem SR650 V3 nodes running ESXi 8.0 Update 3f.

• vmkwarning.log entries showing latency spikes (e.g., Device performance has deteriorated. I/O latency increased from average value of 179 microseconds to 3838 microseconds).
• Latency spikes observed in devicelatencyhisto counts up to 30ms.
• No other software errors or log abnormalities observed.

Environment

VMware ESXi 8.x

Lenovo ThinkSystem SR650 V3

Cause

The Memory Mapped I/O Configuration Base (MM Config Base) setting in the BIOS/UEFI of the Lenovo ThinkSystem SR650 V3 hardware is incorrectly configured or set to a non-optimal value for ESXi.

Resolution

Reach out to Lenovo Technical Support 

To resolve the performance issues, align the BIOS settings with Lenovo's verified hardware tuning guidelines for ESXi:

1. Reboot the ESXi host and enter the BIOS/UEFI Setup.
2. Navigate to System Settings → Devices and I/O Ports → MM Config Base.
3. Change the MM Config Base setting to 3GB.
4. Save the settings and reboot the host.

Setting MMConfigBase = 3GB ensures correct Memory Mapped I/O resource allocation for PCIe storage controllers, preventing device-level I/O latency degradation. Post-reconfiguration, monitor the environment to ensure latency returns to baseline levels.

Additional Optimization: For environments using Round Robin Path Selection Policy (PSP), consider changing the IOPS limit from the default 1000 to 1 to enhance storage path utilization.

 

Additional Information

Lenovo Official VMware ESXi Tuning Guidelines (Article HT115952)

For further assistance, Contact Broadcom support