VirtualMachine.reconfigure task takes 30 minutes to complete

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• This usually completes immediately, but the task of reconfiguring the virtual machine takes about 30 minutes to complete.
  
  
• The following log entries will appear in hostd.log:
  YYYY-MM-DDTHH:MM:SS Wa(164) Hostd[########]: [Originator@6876 sub=RateLimiter opID=######## sid=######## user=vpxuser] Request rejected, 1003 similar already pending, expected to take over ######## minutes (0 48606 0 1033 1033).
  YYYY-MM-DDTHH:MM:SS In(166) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi opID=######## sid=######## user=vpxuser] Throw vmodl.fault.SystemError
  YYYY-MM-DDTHH:MM:SS In(166) Hostd[#######]: [Originator@6876 sub=Solo.Vmomi opID=######## sid=######## user=vpxuser] Result:
  YYYY-MM-DDTHH:MM:SS In(166) Hostd[#######]: --> (vmodl.fault.SystemError) {
  YYYY-MM-DDTHH:MM:SS In(166) Hostd[#######]: -->    reason = "Too many outstanding operations",
  YYYY-MM-DDTHH:MM:SS In(166) Hostd[#######]: -->    msg = "",
  YYYY-MM-DDTHH:MM:SS In(166) Hostd[#######]: --> }
  
  
• The following log entries will appear in vpxa.log:
  YYYY-MM-DDTHH:MM:SS In(166) Vpxa[########]: [Originator@6876 sub=vpxLro opID=########-####-####-####-############;activity=urn:uuid:########-####-####-####-############-##-##-#] [VpxLRO] -- BEGIN task-####### -- vm-### -- vim.VirtualMachine.reconfigure -- ########-####-####-####-############
  ...
  
  YYYY-MM-DDTHH:MM:SS In(166) Vpxa[########]: [Originator@6876 sub=IO.Http opID=########-####-####-####-############;activity=urn:uuid:########-####-####-####-############-##-##-#] Terminating user agent request; <io_obj p:0x############, h:17, <TCP '###.#.#.# : #####'>, <TCP '###.#.#.# : ####'>>, N7Vmacore16TimeoutExceptionE(Operation timed out)
  ...
  
  YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[########]: [Originator@6876 sub=hostdds opID=########-####-####-####-############;activity=urn:uuid:########-####-####-####-############-##-##-#] Exception during datastore refresh: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
  
  
• When using the vim-cmd vimsvc/task_list command, old tasks such as storage refresh/rescans remain
  ex.
  (ManagedObjectReference) [
  'vim.Task:haTask-ha-host-vim.host.StorageSystem.refresh-########',
  'vim.Task:haTask-ha-host-vim.host.StorageSystem.rescanAllHba-########',...

Environment

VMware vCenter Server
VMware vSphere ESXi

Cause

This is caused by old tasks remaining uncompleted on the ESXi host.

Resolution

Workaround:
To resolve the issue, restart hostd to clear any remaining tasks.
/etc/init.d/hostd restart

If the task is not cleared even after restarting hostd, Restart the ESXi host.

Additional Information

A general system error occurred: too many outstanding operations, and the ESXi host can eventually enter a non-responsive state.

VirtualMachine.reconfigure タスクの完了に30分かかる