Host Profile Remediation of ESXi with NFS attached datastores fails . Busy: VSI node (5007:)

Products

• VMware vSphere ESXi

Issue/Introduction

After update of ESXi hosts the attached host profile becomes not compliant.

When remediating the host, the host profile fails.

 

Error observed in the vSphere client when remediating the ESXi.

Error: Operation failed, diagnostics report: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Busy: VSI node (5007:)

 

ESXI Host Logs shows files from storageRM running on the NFS datastores during the remediation

hostd.log

=======

Wa(164) Hostd[xxxxxxx]: [Originator@xxxx sub=Hostsvc.DatastoreSystem opID=xxxxxx-xxxxxx-auto-xxxx-h5:xxxxxxx-xx-xx-xx-xxxx-xxxx sid=xxxxxxxx user=vpxuser] RemoveDatastore: Failed to remove backend for datastore xxxxxxxx.

Wa(164) Hostd[xxxxxxx]: [Originator@xxxx sub=Hostsvc.DatastoreSystem opID=xxxxxx-xxxxxx-auto-xxxx-h5:xxxxxxx-xx-xx-xx-xxxx-xxxx sid=xxxxxxxx user=vpxuser] File descriptors in use:

Wa(164) Hostd[xxxxxxx]: --> [storageRM]    /vmfs/volumes/xxxxxxxxxxx/.iormstats.sf

Wa(164) Hostd[xxxxxxx]: --> Alternative report (all opened file handles, full path is not available)

Wa(164) Hostd[xxxxxxx]: --> volume[xxxxxxxxxxx]/world[storageRM::xxxxxxx] .iormstats.sf

 

Environment

ESXi 8.0

Cause

.iormstats.sf files are open for NFS data stores during the remediation process. 

The .sf files are created by storageRM to store some metadata information and are stopping the NFS datastore from being remounted during the remediation process.

Resolution

To go ahead with the host profile remediation, the IORM service has to be stopped.

There are two methods using which the service can be stopped.

1. In the VC, go to datastore tab, right click on the NFS volume and click on disable sioc.

Additionally, as a validation check to ensure that IORM is disabled, you can check the following command on the ESX hosts in the cluster:

1b. Check the status of storageRM and sdrsInjector

/etc/init.d/storageRM status
/etc/init.d/sdrsInjector status

If status is indicated as "running" do step 2:

2. Stop the service

/etc/init.d/storageRM stop
/etc/init.d/sdrsInjector stop

NOTE: These commands have to be run on _every_ host that is accessing the shared nfs volumes.