Read IO spike every 5 minutes for vSphere 7.0 U1

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Storage array experiences continuous 5 minutes read spike and high CPU utilization.
• Other storage computations like deduplication and compression can be delayed or stalled.

Note: Environment that had the below observations included more than 200 ESXi Host(s) connected to Pure Storage Array.

Environment

VMware vSphere ESXi 7.0.x

Cause

Some changes occurred in vSphere 7.0U1:

1. hostd agent is designed to send  API call every 5 minutes.
2. A new lighter API was added to get required stat.

 

Resolution

Currently there is no resolution.

Workaround:

To work around this issue, we need to increase the VMFS Statistics Intervals. This  can be modified from the config.xml file with below values on each ESXi Host connected to the impacted storage array.

Notes:

• The config.xml file is located in  /etc/vmware/hostd/config.xml
• VMware recommend to change the vmfsStatsIntervalInSecs value to 12 hours 

To do the changes please follow the below steps:

1. Login to the ESXi Host through SSH using root.
2. Run the below command (in one line).

sed -i -e 's/<vmfsStatsIntervalInSecs>.*>/<vmfsStatsIntervalInSecs>43200<\/vmfsStatsIntervalInSecs>/g' /etc/vmware/hostd/config.xml;/etc/init.d/hostd restart

Additional Information

The values of the intervals in minutes/hours:

30 mins  = vmfsStatsIntervalInSecs=1800
1  hour  = vmfsStatsIntervalInSecs=3600
3  hours = vmfsStatsIntervalInSecs=10800
6  hours = vmfsStatsIntervalInSecs=21600
12 hours = vmfsStatsIntervalInSecs=43200

Impact/Risks:
Storage overutilization in case of large amount of ESXi Hosts and large amount of Datastores.