Unexpected datastore IO spikes when the host is in maintenance mode

Products

• VMware vSphere ESXi

Issue/Introduction

1. Unexpected I/O spikes are observed at consistent intervals, exhibiting a recurring pattern in their occurrence.

2. I/O spikes are observed even when the corresponding host is in maintenance mode
3. Other ESXi hosts in the cluster will concurrently see IO spikes to datastores as well.

4. IO spikes stops when below services are stopped in VAMI: 

• VMware vSphere Update Manager
• VMware Service Lifecycle Manager API
• VMware vCenter Life Cycle Management

Cause

Lifecycle management functionality relies on compliance checks. These checks involve various processes, including a Python script called 'precheck.py' which is part of the compliance check framework.

This script performs storage and vmkctl-related commands along with filesystem calls to assess system status. The resulting I/O spike during these checks may be more pronounced in larger clusters with a greater number of datastores.

Resolution

The precheck.py script executes concurrently on all hosts in the cluster, running separately for each ISO on each host during compliance checks.

ISOs are manually uploaded to the lifecycle manager. Once an ISO has passed compliance checks or has been remediated, it can be safely removed from the lifecycle manager. Removing compliant ISOs prevents further I/O activity associated with the precheck.py script. Importantly, other patch and express patch content obtained from VMware's online depot does not trigger the execution of the precheck.py script.

How often do compliance checks need to run ?

Check compliance does not need to be scheduled frequently.
From the release history in Build numbers and versions of VMware ESXi/ESX (2143832), the spacing between two releases is approximately 2 weeks. It is released monthly or once in two months.

A monthly compliance check should be adequate. Additionally, all patch and express patch releases are not based on ISO, which means they will not result in such I/O issues.

Workaround:

1. Check if there are duplicate compliance checks in the VCDB. (Note ID 3 and 4 in the example below):  /opt/vmware/vpostgres/current/bin/psql -d VCDB postgres -c "select scheduledtask_id,name,description from vpx_scheduledtask;"

scheduledtask_id |                       name                       |                        description
------------------+--------------------------------------------------+------------------------------------------------------------
                2 | VMware vSphere Update Manager Check Notification | A predefined scheduled task to check update notifications.
                1 | VMware vSphere Update Manager Update Download    | A predefined scheduled task to download software updates.
		3 | Automatic compliance check DEV_Cluster           | Check for new updates
	        4 | Automatic compliance check                       | 
				
(4 rows)