Troubleshooting Consistent High Load Average on the Linux Virtual Machine

Products

• VMware vSphere ESXi

Issue/Introduction

• Load average is a measurement of the number of processes that currently waiting in the run queue plus the number of processes that are being executed for 1, 5, and 15 minute intervals.
• The higher the load average, the more processes are waiting to run.
• Below is an example of the top output from a Linux VM and from the screenshot we can determine if the number of CPUs on the server is 3, but the load average is between 8.5 and 10, the server is experiencing a high load average.
  
  

Cause

The high load average on the machine can be due to the low number of CPUs configured on the machine or due to high disk I/Ops.

Resolution

To determine the cause of the high Load Average on the virtual machine, we can use "TOP" utility on Linux machines. 

• A load average value is considered high when it’s greater than the number of CPUs the server is configured with.
• Referring to the above example, important parameters to understand here are as below 
  
  

 

1. us, user : Percentage of CPU time running user processes (including root)
2. sy, system : Time running kernel processes. These processes are needed to keep your system alive and operational
3. ni, nice : Time running niced user processes. Niced processes are the ones that have custom priority set with the 'nice' utility.
4. id, idle : Time spent doing nothing
5. wa, IO-wait : Time waiting for I/O completion
6. hi : Time spent servicing hardware interrupts
7. si : Time spent servicing software interrupts
8. st : Time stolen from this VM by the hypervisor

     Correlating the above values to address the high load average 

• Here, the idle time (id) is high (67.7) and the IO-wait percentage (wa) is low (0.0). In this case, the load average is high due to the low number of CPUs configured on the VM. 
• If in case, the I/O wait percentage (wa) was high, then the high load average might be related to disk I/O.