Deploy VM from a template fails with error “A general system error occurred: vDS host error: See FaultCause."

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter reports events similar to:
  • A general system error occurred: vDS host error: See FaultCause.
  • An error occurred while communicating with the remote host. Network configuration change disconnected the host '<hostname>' from vCenter server and has been rolled back.
     
• In vpxd.log you see entries similar to:
  
  YYYY-MM-DDTHH:MM:SS error vpxd[10556] [Originator@6876 sub=vpxLro opID=ProvisioningWizard-add-3186772-ngc:70064932-b1-02] [VpxLRO] Unexpected Exception: class Vmomi::Fault::SystemError::Exception(vmodl.fault.SystemError)
  YYYY-MM-DDTHH:MM:SS info vpxd[10556] [Originator@6876 sub=vpxLro opID=ProvisioningWizard-add-3186772-ngc:70064932-b1-02] [VpxLRO] -- FINISH lro-7682434
  YYYY-MM-DDTHH:MM:SS info vpxd[10556] [Originator@6876 sub=Default opID=ProvisioningWizard-add-3186772-ngc:70064932-b1-02] [VpxLRO] -- ERROR lro-7682434 --  -- VmprovWorkflow: vmodl.fault.SystemError:
  --> Result:
  --> (vmodl.fault.SystemError) {
  -->    faultCause = (vmodl.fault.HostCommunication) {
  -->       faultCause = (vim.fault.NetworkDisruptedAndConfigRolledBack) {
  -->          faultCause = (vmodl.MethodFault) null,
  -->          faultMessage = <unset>,
  -->          host = "<Hostname>"
  -->          msg = ""
  -->       },
  -->       faultMessage = (vmodl.LocalizableMessage) [
  -->          (vmodl.LocalizableMessage) {
  -->             key = "com.vmware.vim.vpxd.network.networkDisruptedAndConfigRolledBack.label",
  -->             arg = (vmodl.KeyAnyValue) [
  -->                (vmodl.KeyAnyValue) {
  -->                   key = "host",
  -->                   value = "<hostname>"
  -->                }
  -->             ],
  -->             message = <unset>
  -->          }
  -->       ]
  -->       msg = ""
  -->    },
  -->    faultMessage = (vmodl.LocalizableMessage) [
  -->       (vmodl.LocalizableMessage) {
  -->          key = "com.vmware.vim.vpxd.dvs.portPushFail.label",
  -->          arg = (vmodl.KeyAnyValue) [
  -->             (vmodl.KeyAnyValue) {
  -->                key = "port",
  -->                value = "1457"
  -->             },
  -->             (vmodl.KeyAnyValue) {
  -->                key = "dvs",
  -->                value = "<port name>"
  -->             },
  -->             (vmodl.KeyAnyValue) {
  -->                key = "host",
  -->                value = "<host name>"
  -->             }
  -->          ],
  -->          message = <unset>
  -->       }
  -->    ],
  -->    reason ="
  -->    msg = ""
   YYYY-MM-DDTHH:MM:SS info hostd[10640B70] [Originator@6876 sub=Solo.Vmomi opID=3949c184-636d user=vpxuser] Throw vmodl.fault.SystemError
   YYYY-MM-DDTHH:MM:SS info hostd[10640B70] [Originator@6876 sub=Solo.Vmomi opID=3949c184-636d user=vpxuser] Result

--> (vmodl.fault.SystemError) {

-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Transaction has rolled back on the host."
-->    msg = ""
 

 

• Host sync takes more than 30 sec with VsanSimsStubImpl. In hostd.log you see entries similar to:

YYYY-MM-DDTHH:MM:SS info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystemEx.GetVsanNics
YYYY-MM-DDTHH:MM:SS info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystemEx.GetVsanNics
YYYY-MM-DDTHH:MM:SS info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystemEx.GetVsanNics
YYYY-MM-DDTHH:MM:SS info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystemEx.GetVsanNics
YYYY-MM-DDTHH:MM:SS info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystem.GetConfig
YYYY-MM-DDTHH:MM:SS info hostd[122C6B70] [Originator@6876 sub=Libs opID=65df636b] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_spm_1.0.0, localId: spm
YYYY-MM-DDTHH:MM:SS info hostd[122C6B70] [Originator@6876 sub=Libs opID=65df636b] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_vmwarevmcrypt_1.0.0, localId: vmwarevmcrypt
YYYY-MM-DDTHH:MM:SS info hostd[122C6B70] [Originator@6876 sub=Libs opID=65df636b] PluginLdr_Load: Loaded plugin 'libvmiof-disk-spm.so' from '/usr/lib/vmware/plugin/libvmiof-disk-spm.so'
YYYY-MM-DDTHH:MM:SS info hostd[122C6B70] [Originator@6876 sub=Libs opID=65df636b] PluginLdr_Load: Loaded plugin 'libvmiof-disk-vmwarevmcrypt.so' from '/usr/lib/vmware/plugin/libvmiof-disk-vmwarevmcrypt.so'
YYYY-MM-DDTHH:MM:SS warning hostd[122C6B70] [Originator@6876 sub=PropertyProvider opID=65df636b] It took 30095502 microseconds to get property config for vim.HostSystem:ha-host
YYYY-MM-DDTHH:MM:SS warning hostd[122C6B70] [Originator@6876 sub=PropertyCollector opID=65df636b]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Cause

This issue occurs due to VsanSimsStubImpl taking more time to complete during host sync and causing vCenter Service to timeout.

Resolution

• This issue is resolved in ESXi 6.5 Patch Release ESXi650-201811002, available at Broadcom Downloads

Workaround:

• To workaround this issue, increase config.vpxd.network.rollbackTimeout:

1. On the Manage tab of a vCenter Server instance, click Settings.
2. Select Advanced Settings and click Edit.
3. If the property is not present, add the config.vpxd.network.rollbackTimeout parameter to the settings.
4. Type a new value, start with 90 seconds, for the config.vpxd.network.rollbackTimeout parameter
5. Click OK.