Error: "A general system error occurred: Launch failure YYYY-MM-DDTHH:MM:SS.451039Z Transport (VMDB) error -45: Failed to connect to peer process" when vMotion failed

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

The target VM has more than 1 disk and is stored in different datastore.

When vMotion fails, the task will show the error: A general system error occurred: Launch failure YYYY-MM-DDTHH:MM:SS.451039Z Transport (VMDB) error -45: Failed to connect to peer process

The ESXi server /var/log/vmkernel.log file frequently displays warnings similar to:

YYYY-MM-DDTHH:MM:SS.754Z verbose hostd[486001941] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<Datastore_UUID>/<VM_Name>/<VM_Name>.vmx] VMotionStatusCb: Firing ResolveCb
YYYY-MM-DDTHH:MM:SS.754Z info hostd[486001941] [Originator@6876 sub=Vcsvc.VMotionSrc.1781924420074684338] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION
YYYY-MM-DDTHH:MM:SS.754Z info hostd[486001941] [Originator@6876 sub=Vcsvc.VMotionSrc.1781924420074684338] ResolveCb: Failed with fault: (vim.fault.GenericVmConfigFault) {
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "msg.vigor.operationCancelled",
-->          message = "The operation was cancelled by the user.",
-->       }
-->    ],
-->    reason = "The operation was cancelled by the user.",
-->    msg = "The operation was cancelled by the user.
--> "
--> }
YYYY-MM-DDTHH:MM:SS.755Z info hostd[486001941] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<Datastore_UUID>/<VM_Name>/<VM_Name>.vmx] State Transition (VM_STATE_EMIGRATING -> VM_STATE_ON)
YYYY-MM-DDTHH:MM:SS.764Z info hostd[486001941] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<Datastore_UUID>/<VM_Name>/<VM_Name>.vmx] Send config update invoked
--> [context]zKq7AVICAgAAAKjoRgEUaG9zdGQAAC2hQmxpYnZtYWNvcmUuc28AAOJnHQAt0RsBcyvBaG9zdGQAAQxqwQGFUM8BMpnPARGeXwEloV8B8qpfAcu0zwEIRs8B4gnCARNThQEE9s0AzMctAFQDLgACET8CO30AbGlicHRocmVhZC5zby4wAANt0Q5saWJjLnNvLjYA[/context]
YYYY-MM-DDTHH:MM:SS.808Z verbose hostd[486001941] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<Datastore_UUID>/<VM_Name>/<VM_Name>.vmx] Time to gather config: 41 (msecs)
YYYY-MM-DDTHH:MM:SS.817Z info hostd[486001941] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<Datastore_UUID>/<VM_Name>/<VM_Name>.vmx] VMotion cleanup completed

The vCenter server /var/log/vmware/vpxd/vpxd.log file frequently displays warnings similar to:

YYYY-MM-DDTHH:MM:SS info vpxd[10976] [Originator@6876 sub=vmomi.soapStub[779251] opID=m3760w3v-13085982-auto-7sh7j-h5:71038810-45-01] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007ff0c4722f38, h:30, <TCP '<IP> : 59708'>, <TCP '<IP>  : 443'>>), /vpxa>, method: initiateDestination; code: 500(Internal Server Error); fault: (vmodl.fault.SystemError) {
-->    faultCause = (vim.fault.GenericVmConfigFault) {
-->       faultCause = (vmodl.MethodFault) null,
-->       faultMessage = (vmodl.LocalizableMessage) [
-->          (vmodl.LocalizableMessage) {
-->             key = "msg.vigor.transport.vmdb",
-->             arg = (vmodl.KeyAnyValue) [
-->                (vmodl.KeyAnyValue) {
-->                   key = "1",
-->                   value = "-45"
-->                },
-->                (vmodl.KeyAnyValue) {
-->                   key = "2",
-->                   value = "Failed to connect to peer process"
-->                }
-->             ],
-->             message = "Transport (VMDB) error -45: Failed to connect to peer process."
-->          },
-->          (vmodl.LocalizableMessage) {
-->             key = "faultTime",
-->             arg = <unset>,
-->             message = "YYYY-MM-DDTHH:MM:SS"
-->          }
-->       ],
-->       reason = "Transport (VMDB) error -45: Failed to connect to peer process."
-->       msg = "Transport (VMDB) error -45: Failed to connect to peer process."
-->    },
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "msg.vigor.transport.vmdb",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "1",
-->                value = "-45"
-->             },
-->             (vmodl.KeyAnyValue) {
-->                key = "2",
-->                value = "Failed to connect to peer process"
-->             }
-->          ],
-->          message = "Transport (VMDB) error -45: Failed to connect to peer process."
-->       },
-->       (vmodl.LocalizableMessage) {
-->          key = "faultTime",
-->          arg = <unset>,
-->          message = "YYYY-MM-DDTHH:MM:SS"
-->       }
-->    ],
-->    reason = "Launch failure"
-->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007ff0c4722f38, h:30, <TCP '<IP> : 59708'>, <TCP '<IP> : 443'>>), /vpxa>]: initiateDestination
--> Received SOAP response fault from [<<io_obj p:0x000000156cc4c6c8, h:18, <TCP '127.0.0.1 : 21833'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: initiateDestination
--> A general system error occurred: Launch failure"
--> }
YYYY-MM-DDTHH:MM:SS error vpxd[10976] [Originator@6876 sub=VmProv opID=m3760w3v-13085982-auto-7sh7j-h5:71038810-45-01] Get exception while executing action vpx.vmprov.InitiateDestination: N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError
--> )
--> 

Environment

vCenter Server 7.0.3

VMware vSphere ESXi 7.0.3

Cause

iSCSI used physical Nic (vmnic) is incompatible.

Resolution

To address this issue, please contact Server Vendor Support to make physical Nic (vmnic) in compatible version.

workaround:

Perform storage vMotion for target VM, make its disks into the same Datastore same VM folder.

Then, perform vMotion for the target VM.