Storage vMotion initiated from Cloud Director causes vCenter vpxd service to crash.

Products

• VMware vCenter Server

Issue/Introduction

In some environments and rare circumstance the vpxd service on a vCenter server may crash after triggering a storage vMotion workflow from Cloud Director.

• The error "Unexpected status code: 503" appears in the VCD UI.

• Logs from /opt/vmware/vcloud-director/logs/vcloud-container-debug.log include:

  com.vmware.ssdc.util.LMException: Unexpected status code: 503 at com.vmware.ssdc.util.LMException.wrap(LMException.java:135) at com.vmware.ssdc.library.ExceptionFactory.createFromMultiple(ExceptionFactory.java:32) at com.vmware.vcloud.vdc.impl.ComposeRecomposeVAppActivity$FinalizeRecomposePhase.invoke(ComposeRecomposeVAppActivity.java:1592) Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503

• Manual trigger of storage vMotion from vCenter directly is unaffected

Environment

vCenter 7x
vCenter 8x
VMware Cloud Director

Cause

VPXD is crashing while executing the RecommendDatastores call to get the recommendation disk placement for a virtual machine with a large size.

Example:
-A virtual machine with 17 disks
-First 16 disks of the virtual machine can be placed on datastore A
-Datastore A does not have enough space for the 17th disk
-Placement algorithm runs again and vpxd crashes.

Resolution

VMware By Broadcom Engineering team are aware of the issue and planning a fix for future release.

Workaround:

Initiate the storage vmotion from vCenter directly.