Diego BBS Instance with High CPU or High Load Average Seen

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

High CPU usage of 99%, as well as a high load average when using the command "bosh vms --vitals" was seen, as shown below. Note: Output of BOSH VMs was also reduced.

+-----------------------------------------------------------------------------------+---------+-----+------------------------------------------+-------------+-----------------------+------+------+------+--------------+------------+------------+------------+------------+
| VM                                                                                | State   | AZ  | VM Type                                  | IPs         |         Load          | CPU  | CPU  | CPU  | Memory Usage | Swap Usage | System     | Ephemeral  | Persistent |
|                                                                                   |         |     |                                          |             | (avg01, avg05, avg15) | User | Sys  | Wait |              |            | Disk Usage | Disk Usage | Disk Usage |
+-----------------------------------------------------------------------------------+---------+-----+------------------------------------------+-------------+-----------------------+------+------+------+--------------+------------+------------+------------+------------+
| diego_database-partition-a1684d9111ac3fc8826f/0                                   | running | n/a | diego_database-partition-a1..3fc7726f    | 10.91.56.36  | 2.92, 2.03, 1.95     |99.7% | 1.0% | 0.3% | 14% (138.6M) | 0% (60.0K) | 44%        | 24%        | 30%        |

This might cause possible performance issues. 

 

Environment

Resolution

For CPU Load averages (avg01, avg05, avg15) it is important to understand the 3 numbers and what they represent, which are periods of time for one, five, and fifteen-minute averages. The lower the number, the better it is. High load could indicate a problem or that the VM is overloaded. To determine if Load and CPU indicate a possible performance issue, we need to look at how the CPU processes work (load). For a single core, we have one queue to process work. So if load is 0.00, that means there is no work to process. If there is load of 1.00, that means we are working at full capacity. If it is greater than 1.00, for example 2.00, then we have enough work for 2 CPUs (or 2 queues).

In this case, if you increase the number of CPUs for the Diego database partition on the resource config page in Ops Manager, you should see the Load average and CPU use decrease.

If increasing the CPU does not help, then looking at the processes running on the VM is the next step. Factors affecting the Load average include CPU use and waiting for CPU, disk IO being performed, and network IO. If you encounter a situation where increasing the CPU does not help, please contact Pivotal support.