Cannot power on vCloud Director VMs with guest customization enabled
book
Article ID: 325481
calendar_today
Updated On:
Products
VMware Cloud Director
Issue/Introduction
Symptoms:
When ESXi hosts in Provider cluster are in standby mode, you experience these symptoms:
When powering on a virtual machine in vCloud Director which has guest customization enabled, the virtual machine is placed on the active host which is not in Standby Mode but the task to place the CAB file for customization seems to try to go through one of the hosts which are in Standby Mode.
In vCloud Director, the ESXi hosts are shown with two System Alerts:
The ESXi host is unavailable
The ESXi is in Standby Mode
You see error trace similar to:
2014-08-08 15:45:21,609 | DEBUG | backend-activity-pool-319 | CJob | updateFailedJob(com.vmware.ssdc.util.LMException) with locale=null | vcd=<VCD_UUID>,task=<TASK_UUID> activity=urn:uuid:<TASK_UUID> com.vmware.ssdc.util.LMException: Unable to start vApp "<VAPP_NAME>". at com.vmware.ssdc.backend.AbstractVAppImpl.deploySubset(AbstractVAppImpl.java:526) at com.vmware.ssdc.backend.services.impl.VAppDeploymentManagerImpl.deployVm(VAppDeploymentManagerImpl.java:398) at com.vmware.vcloud.vapp.impl.VAppServiceImpl.deployTask(VAppServiceImpl.java:1894) at com.vmware.vcloud.vapp.impl.VAppServiceImpl.executeTask(VAppServiceImpl.java:691) at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:427) at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:421) at com.vmware.vcloud.backendbase.management.system.TaskActivity$SecurityContextTemplate.executeForOrgAndUser(TaskActivity.java:674) at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:421) at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invoke(TaskActivity.java:354) at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:109) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: com.vmware.ssdc.util.LMException: HTTP NFC Error. Cannot deploy package from /opt/vmware/vcloud-director/tmp/<CAB_FILENAME> to '<VM_FILE_PATH>/<CAB_FILENAME>'. at com.vmware.vcloud.customization.impl.DeploymentPackage.DeployPackage(DeploymentPackage.java:251) at com.vmware.vcloud.customization.impl.CustomizationServiceImpl.pushDeploymentPackage(CustomizationServiceImpl.java:196) at com.vmware.vcloud.customization.impl.VmGuestCustomizationActivity$PushDeploymentPackagePhase.invoke(VmGuestCustomizationActivity.java:80) ... 7 more Caused by: com.vmware.vcloud.fabric.storage.copy.CompleteFailureException at com.vmware.vcloud.fabric.storage.copy.impl.HttpFileCopyService.handleActivityFuture(HttpFileCopyService.java:203) at com.vmware.vcloud.fabric.storage.copy.impl.HttpFileCopyService.copyToDatastore(HttpFileCopyService.java:123) at sun.reflect.GeneratedMethodAccessor6227.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319) at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58) at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56) at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy736.copyToDatastore(Unknown Source) at com.vmware.vcloud.customization.impl.DeploymentPackage.DeployPackage(DeploymentPackage.java:248) ... 9 more Caused by: com.vmware.ssdc.util.LMException: java.net.NoRouteToHostException: No route to host
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Resolution
To resolve the issue, manually disable the ESXi hosts which are in Standby mode from vCloud Director. vCloud Director now no longer tries to use them to place the CAB file and the virtual machine powers on and customize.