Deploying Guest Introspection virtual machines through EAM fails

Products

• VMware NSX Networking
• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Deploying Guest Introspection virtual machines through EAM fails.
• Cloning virtual machines are stuck at 29%.

• In the EAM logs, you see entries similar to:

  2017-08-21T13:28:34.283-04:00 | INFO | host-1450-1 | DeployVmJob.java | 389 | Making full clone of VM: ManagedObjectReference: type = VirtualMachine, value = vm-28135, serverGuid = <UUID>
  2017-08-21T13:28:34.283-04:00 | INFO | host-1450-1 | DeployVmJob.java | 389 | Making full clone of VM: ManagedObjectReference: type = VirtualMachine, value = vm-28135, serverGuid = <UUID>
  2017-08-21T13:28:34.341-04:00 | WARN | vim-async-0 | ChangeStream.java | 208 | No change set supplied! Missing set is null
  2017-08-21T14:08:49.145-04:00 | WARN | host-1450-1 | DeployVmJob.java | 244 | VM cloning failedcom.vmware.eam.EamAppException: Failed to clone VM at com.vmware.eam.vc.vm.impl.AbstractVmCloner.cloneVm(AbstractVmCloner.java:177) at com.vmware.eam.vc.vm.impl.AbstractVmCloner.deploy(AbstractVmCloner.java:91) at com.vmware.eam.job.DeployVmJob.performVmCloning(DeployVmJob.java:451) at com.vmware.eam.job.DeployVmJob.cloneExistingVm(DeployVmJob.java:392) at com.vmware.eam.job.DeployVmJob.launchNewVm(DeployVmJob.java:241) at com.vmware.eam.job.DeployVmJob.doJob(DeployVmJob.java:209) at com.vmware.eam.job.DeployVmJob.call(DeployVmJob.java:156) at com.vmware.eam.job.DeployVmJob.call(DeployVmJob.java:108) at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:35) at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:52)

• In the vpxd logs, you see entries similar to:

  2017-08-21T14:08:46.105-04:00 warning vpxd[16900] [Originator@6876 sub=vpxLro] [VpxLRO] Timeout waiting on updates for task-54
  2017-08-21T14:08:46.105-04:00 warning vpxd[16900] [Originator@6876 sub=vpxLro] [VpxLRO] Timeout waiting on updates for task-54
  2017-08-21T14:08:46.122-04:00 error vpxd[04468] [Originator@6876 sub=VmProv opID=17ec1d3e-01] Failed to track task vim.Task:task-54 on host vim.HostSystem:host-1351: vim.fault.Timedout
  --> backtrace:
  --> [backtrace begin] product: VMware VirtualCenter, version: 6.5.0, build: build-5705665, tag: vpxd, cpu: x86_64, os: windows, buildType: release
  --> backtrace[00] vmacore.dll[0x001F4ABA]
  --> backtrace[01] vmacore.dll[0x000680F0]
  --> backtrace[02] vmacore.dll[0x0006AA7E]
  .....
  omitted
  .....
  2017-08-21T14:08:49.066-04:00 info vpxd[04468] [Originator@6876 sub=vpxLro opID=17ec1d3e-01] [VpxLRO] -- FINISH lro-10631155
  2017-08-21T14:08:49.066-04:00 info vpxd[04468] [Originator@6876 sub=Default opID=17ec1d3e-01] [VpxLRO] -- ERROR lro-10631155 -- -- VmprovWorkflow: vim.fault.Timedout:
  --> Result:
  --> (vim.fault.Timedout) {
  --> faultCause = (vmodl.MethodFault) null,
  --> faultMessage = <unset>
  --> msg = ""
  --> }
  --> Args:
  -->
  2017-08-21T14:08:49.076-04:00 warning vpxd[04092] [Originator@6876 sub=VpxProfiler opID=17ec1d3e-01-TaskLoop-44463581] TaskLoop [TotalTime] took 2403000 ms
  2017-08-21T14:08:49.076-04:00 info vpxd[04468] [Originator@6876 sub=vpxLro opID=17ec1d3e] [VpxLRO] -- FINISH task-568746
  2017-08-21T14:08:49.076-04:00 info vpxd[04468] [Originator@6876 sub=Default opID=17ec1d3e] [VpxLRO] -- ERROR task-568746 -- vm-28135 -- vim.VirtualMachine.clone: vim.fault.Timedout:
  --> Result:

  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.5
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.5.x

Cause

This issue occurs because the Agent Settings on the ESXi host is set to a datastore which no longer exists.

Note: This issue will only be seen if the Service Deployment uses "specified on host" for the datastore deployment option.

Resolution

 

To resolve this issue, change the Agent settings on each of the ESXi host in the cluster to a different datastore. Then change the option back to the desired value.