[INTERNAL] ESXi compliance scan / remediation takes a long time.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

Compliance scan / remediation takes a long time on hosts which have a large number of data stores attached to them. 

From the vua logs on the host, we can see that the call to forkexec precheck.py takes a long time to run.

For example:

• 2023-03-13T17:59:18.407Z info vua[2103400] [Originator@6876 sub=VUA] Invoking: "/tmp/vuaScript-s6zWcx/precheck.py --ip=10.128.10.18"

2023-03-13T17:59:18.409Z info vua[2103400] [Originator@6876 sub=SysCommandPosix] ForkExec(/tmp/vuaScript-s6zWcx/precheck.py) 2103413

2023-03-13T18:09:29.841Z info vua[2103411] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 9, idle: 1

2023-03-13T18:09:29.841Z info vua[2103784] [Originator@6876 sub=ThreadPool] Entering worker thread loop

In the above case, the next call after the ForkExec precheck.py is happening after 10 minutes.

• 2023-05-16T11:41:50.298Z info vua[87520504] [Originator@6876 sub=VUA] Invoking: "/tmp/vuaScript-GM91hP/precheck.py --ip=xxx.xx.xx.xx"

2023-05-16T11:41:50.301Z info vua[87520504] [Originator@6876 sub=SysCommandPosix] ForkExec(/tmp/vuaScript-GM91hP/precheck.py) 87520529

2023-05-16T12:06:12.517Z info vua[87520510] [Originator@6876 sub=ThreadPool] Spawning additional worker - allocated: 9, idle: 1

2023-05-16T12:06:12.517Z info vua[87525286] [Originator@6876 sub=ThreadPool] Entering worker thread loop

The next statement after ForkExec is after 25 minutes.

 

 

Environment

VMware vSphere ESXi 7.0.3

Cause

The compliance scans that run on the hosts (either as an independent operation or as part of remediation) have upgrade prechecks which list out all the attached volumes, the free space on each of these, their versions, etc. If a host is attached to a large number of data stores, each with a large capacity, this operation can take a lot of time. Also, this time is multiplied by the number of baselines attached to the cluster / host.

Resolution

This issue has been fixed in 7.0.3 P08

Workaround:

If it is feasible, please detach all the attached data store volumes temporarily and perform the required scans or remediation. Once these operations are completed, the volumes can be re-attached.