Deploy VM from a template fails with error “A general system error occurred: vDS host error: See FaultCause."

Products

• VMware

Issue/Introduction

Symptoms:

• vCenter reports events similar to:
  • A general system error occurred: vDS host error: See FaultCause.
  • An error occurred while communicating with the remote host. Network configuration change disconnected the host '<hostname>' from vCenter server and has been rolled back.
     
• In vpxd.log you see entries similar to:
  
  2018-09-26T11:56:14.225+05:30 error vpxd[10556] [Originator@6876 sub=vpxLro opID=ProvisioningWizard-add-3186772-ngc:70064932-b1-02] [VpxLRO] Unexpected Exception: class Vmomi::Fault::SystemError::Exception(vmodl.fault.SystemError)
  2018-09-26T11:56:14.234+05:30 info vpxd[10556] [Originator@6876 sub=vpxLro opID=ProvisioningWizard-add-3186772-ngc:70064932-b1-02] [VpxLRO] -- FINISH lro-7682434
  2018-09-26T11:56:14.234+05:30 info vpxd[10556] [Originator@6876 sub=Default opID=ProvisioningWizard-add-3186772-ngc:70064932-b1-02] [VpxLRO] -- ERROR lro-7682434 --  -- VmprovWorkflow: vmodl.fault.SystemError:
  --> Result:
  --> (vmodl.fault.SystemError) {
  -->    faultCause = (vmodl.fault.HostCommunication) {
  -->       faultCause = (vim.fault.NetworkDisruptedAndConfigRolledBack) {
  -->          faultCause = (vmodl.MethodFault) null,
  -->          faultMessage = <unset>,
  -->          host = "<Hostname>"
  -->          msg = ""
  -->       },
  -->       faultMessage = (vmodl.LocalizableMessage) [
  -->          (vmodl.LocalizableMessage) {
  -->             key = "com.vmware.vim.vpxd.network.networkDisruptedAndConfigRolledBack.label",
  -->             arg = (vmodl.KeyAnyValue) [
  -->                (vmodl.KeyAnyValue) {
  -->                   key = "host",
  -->                   value = "<hostname>"
  -->                }
  -->             ],
  -->             message = <unset>
  -->          }
  -->       ]
  -->       msg = ""
  -->    },
  -->    faultMessage = (vmodl.LocalizableMessage) [
  -->       (vmodl.LocalizableMessage) {
  -->          key = "com.vmware.vim.vpxd.dvs.portPushFail.label",
  -->          arg = (vmodl.KeyAnyValue) [
  -->             (vmodl.KeyAnyValue) {
  -->                key = "port",
  -->                value = "1457"
  -->             },
  -->             (vmodl.KeyAnyValue) {
  -->                key = "dvs",
  -->                value = "<port name>"
  -->             },
  -->             (vmodl.KeyAnyValue) {
  -->                key = "host",
  -->                value = "<host name>"
  -->             }
  -->          ],
  -->          message = <unset>
  -->       }
  -->    ],
  -->    reason ="
  -->    msg = ""
   2018-09-26T06:26:13.615Z info hostd[10640B70] [Originator@6876 sub=Solo.Vmomi opID=3949c184-636d user=vpxuser] Throw vmodl.fault.SystemError
   2018-09-26T06:26:13.615Z info hostd[10640B70] [Originator@6876 sub=Solo.Vmomi opID=3949c184-636d user=vpxuser] Result

--> (vmodl.fault.SystemError) {

-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Transaction has rolled back on the host."
-->    msg = ""
 

 

• Host sync takes more than 30 sec with VsanSimsStubImpl. In hostd.log you see entries similar to:

2018-09-26T06:25:43.548Z info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystemEx.GetVsanNics
2018-09-26T06:25:43.549Z info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystemEx.GetVsanNics
2018-09-26T06:26:13.610Z info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystemEx.GetVsanNics
2018-09-26T06:26:13.610Z info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystemEx.GetVsanNics
2018-09-26T06:26:13.612Z info hostd[122C6B70] [Originator@6876 sub=VsanSimsStubImpl opID=65df636b] Calling vim.host.VsanSystem.GetConfig
2018-09-26T06:26:13.612Z info hostd[122C6B70] [Originator@6876 sub=Libs opID=65df636b] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_spm_1.0.0, localId: spm
2018-09-26T06:26:13.612Z info hostd[122C6B70] [Originator@6876 sub=Libs opID=65df636b] IOFilterInfoImpl: Inbox-IOFilter Id: VMW_vmwarevmcrypt_1.0.0, localId: vmwarevmcrypt
2018-09-26T06:26:13.613Z info hostd[122C6B70] [Originator@6876 sub=Libs opID=65df636b] PluginLdr_Load: Loaded plugin 'libvmiof-disk-spm.so' from '/usr/lib/vmware/plugin/libvmiof-disk-spm.so'
2018-09-26T06:26:13.614Z info hostd[122C6B70] [Originator@6876 sub=Libs opID=65df636b] PluginLdr_Load: Loaded plugin 'libvmiof-disk-vmwarevmcrypt.so' from '/usr/lib/vmware/plugin/libvmiof-disk-vmwarevmcrypt.so'
2018-09-26T06:26:13.615Z warning hostd[122C6B70] [Originator@6876 sub=PropertyProvider opID=65df636b] It took 30095502 microseconds to get property config for vim.HostSystem:ha-host
2018-09-26T06:26:13.623Z warning hostd[122C6B70] [Originator@6876 sub=PropertyCollector opID=65df636b]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs due to VsanSimsStubImpl taking more time to complete during host sync and causing vCenter Service to timeout.

Resolution

• This issue is resolved in ESXi 6.5 Patch Release ESXi650-201811002, available at VMware Patch Downloads.

Workaround:

• To workaround this issue, increase config.vpxd.network.rollbackTimeout:

1. On the Manage tab of a vCenter Server instance, click Settings.
2. Select Advanced Settings and click Edit.
3. If the property is not present, add the config.vpxd.network.rollbackTimeout parameter to the settings.
4. Type a new value, start with 90 seconds, for the config.vpxd.network.rollbackTimeout parameter
5. Click OK.
 

Additional Information

VM reconfiguration fails while performing a test recovery with SRM