Cannot power on vCloud Director VMs with guest customization enabled

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

When ESXi hosts in Provider cluster are in standby mode, you experience these symptoms:

• When powering on a virtual machine in vCloud Director which has guest customization enabled, the virtual machine is placed on the active host which is not in Standby Mode but the task to place the CAB file for customization seems to try to go through one of the hosts which are in Standby Mode.
• In vCloud Director, the ESXi hosts are shown with two System Alerts:
   
  • The ESXi host is unavailable
  • The ESXi is in Standby Mode
     
• You see error trace similar to:
  
  2014-08-08 15:45:21,609 | DEBUG | backend-activity-pool-319 | CJob | updateFailedJob(com.vmware.ssdc.util.LMException) with locale=null | vcd=<VCD_UUID>,task=<TASK_UUID> activity=urn:uuid:<TASK_UUID>
  com.vmware.ssdc.util.LMException: Unable to start vApp "<VAPP_NAME>".
  at com.vmware.ssdc.backend.AbstractVAppImpl.deploySubset(AbstractVAppImpl.java:526)
  at com.vmware.ssdc.backend.services.impl.VAppDeploymentManagerImpl.deployVm(VAppDeploymentManagerImpl.java:398)
  at com.vmware.vcloud.vapp.impl.VAppServiceImpl.deployTask(VAppServiceImpl.java:1894)
  at com.vmware.vcloud.vapp.impl.VAppServiceImpl.executeTask(VAppServiceImpl.java:691)
  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:427)
  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase$2.doInSecurityContext(TaskActivity.java:421)
  at com.vmware.vcloud.backendbase.management.system.TaskActivity$SecurityContextTemplate.executeForOrgAndUser(TaskActivity.java:674)
  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.execute(TaskActivity.java:421)
  at com.vmware.vcloud.backendbase.management.system.TaskActivity$ExecutePhase.invoke(TaskActivity.java:354)
  at com.vmware.vcloud.activity.executors.ActivityRunner.run(ActivityRunner.java:109)
  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
  at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
  at java.util.concurrent.FutureTask.run(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  at java.lang.Thread.run(Unknown Source)
  Caused by: com.vmware.ssdc.util.LMException: HTTP NFC Error. Cannot deploy package from /opt/vmware/vcloud-director/tmp/<CAB_FILENAME> to '<VM_FILE_PATH>/<CAB_FILENAME>'.
  at com.vmware.vcloud.customization.impl.DeploymentPackage.DeployPackage(DeploymentPackage.java:251)
  at com.vmware.vcloud.customization.impl.CustomizationServiceImpl.pushDeploymentPackage(CustomizationServiceImpl.java:196)
  at com.vmware.vcloud.customization.impl.VmGuestCustomizationActivity$PushDeploymentPackagePhase.invoke(VmGuestCustomizationActivity.java:80)
  ... 7 more
  Caused by: com.vmware.vcloud.fabric.storage.copy.CompleteFailureException
  at com.vmware.vcloud.fabric.storage.copy.impl.HttpFileCopyService.handleActivityFuture(HttpFileCopyService.java:203)
  at com.vmware.vcloud.fabric.storage.copy.impl.HttpFileCopyService.copyToDatastore(HttpFileCopyService.java:123)
  at sun.reflect.GeneratedMethodAccessor6227.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
  at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58)
  at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
  at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56)
  at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
  at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
  at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
  at com.sun.proxy.$Proxy736.copyToDatastore(Unknown Source)
  at com.vmware.vcloud.customization.impl.DeploymentPackage.DeployPackage(DeploymentPackage.java:248)
  ... 9 more
  Caused by: com.vmware.ssdc.util.LMException: java.net.NoRouteToHostException: No route to host
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment. 　

Resolution

To resolve the issue, manually disable the ESXi hosts which are in Standby mode from vCloud Director. vCloud Director now no longer tries to use them to place the CAB file and the virtual machine powers on and customize.