Disabling Storage I/O Control in ESXi hosts fails with the error: Configure Storage I/O Control on Datastore datastore_name The operation is not supported on the object

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Cannot disable Storage I/O Control in ESXi hosts
• Disabling Storage I/O Control in ESXi hosts fails
• You see the error:
  
  Configure Storage I/O Control on Datastore datastore_name The operation is not supported on the object. DOMAIN\username vcenter.domain.com
   
• The vpxd.log file in vCenter Server shows that the ESXi host does not support Storage I/O Control (IORM):
  
  2013-04-24T11:33:37.307-04:00 [07136 error 'vpxdvpxdMoStorageResourceManager' opID=8efbe83a] [VpxDrmStorageResourceManager::ConfigureDatastoreIORM] Failed to configure IORM on datastore datastore-23445 through host esx8.domain.com (host-14659): vmodl.fault.NotSupported
   
• Running the lsof command from the ESXi host CLI shows that the only open file on the VMFS or NFS is .iormstats.sf , but you are unable to unmount the datastore due to message Active IO:
  
  # lsof | grep UUID
  2829 storageRM 12 5 /vmfs/volumes/UUID/.iormstats.sf
• In the /var/log/vpxa.log file, you will see entries similar to,

[Originator@6876 sub=Host] Heartbeat handler detected dead connection for host: host-19914
[Originator@6876 sub=Main] HbrError stack:
[Originator@6876 sub=Main] [0] Exception Vmacore::InvalidStateException: No connection (host=host-19914)
[Originator@6876 sub=Main] [1] Failed to get time (host=host-19914)
2017-09-08T19:56:34.833Z info hbrsrv[7FA06FDB6700] [Originator@6876 sub=Main] [2] Ignored error.
2017-09-08T19:56:34.833Z info hbrsrv[7FA06FDB6700] [Originator@6876 sub=Main] HbrError stack:
2017-09-08T19:56:34.833Z info hbrsrv[7FA06FDB6700] [Originator@6876 sub=Main] [0] Database object not found
2017-09-08T19:56:34.833Z info hbrsrv[7FA06FDB6700] [Originator@6876 sub=Main] [1] Looking up global key in the database guestinfo.hbr.hbrsrv-certificate-revoked/guestinfo-cache
2017-09-08T19:56:34.833Z info hbrsrv[7FA06FDB6700] [Originator@6876 sub=Main] [2] Couldn't read cached value for key guestinfo.hbr.hbrsrv-certificate-revoked
2017-09-08T19:56:34.833Z info hbrsrv[7FA06FDB6700] [Originator@6876 sub=Main] [3] Ignored error.
2017-09-08T19:56:34.833Z info hbrsrv[7FA06FDB6700] [Originator@6876 sub=HostAgentConnection] Host host-19914: restarting with address 10.97.3.127

Cause

During startup, vSphere Replication hbrsrv needs to try to connect to all hosts in the vCenter inventory. If customer has a large environment and a single hosts can not connect to hbrsrv, it will take long time to loop through them. Network firewall may add unnecessary time to HBR service startup.
 
If an ESXi host is removed from the VC inventory while the vSphere Replication hbrsrv is offline that will also interfere with the startup of the service as vSphere Replication hbrsrv still has an ESXi entry for that host in its database.

Resolution

To resolve this issue, upgrade the license of the ESX/ESXi host accessing the datastore to the Enterprise Plus Edition.

 

Alternatively, remove the ESX/ESXi host from vCenter Server.

Workaround:
To workaround this issue, disable SIOC to reduce the event frequency.